CEC: In Review

CEC2006: Too much content not enough time



Next Year:

• href="http://www.evdoinfo.com/EVDO_Products/EVDO_Routers/Kyocera_KR1_EVDO_Router_with_WiFi_20050427264/">Kyocera
  KR1




• It seems that the state of high density WiFi may be
  slightly less than carrier grade, at least for temporary third party
  installations




• The good camera




• Not that the Cannon Point and Shoot is bad, just that I
  found myself wanting the more easily adjustable manual settings of the
  D1x




• The Managed Operations CEC blog/multi media Tiger Team
  (MOCECBMMTT? OK I made that up)

Interactivity(Web2.0):

• It was interesting, and hey I apparently won $200 for the
  most
  useful sms messages. I have yet to be contacted about my prize, maybe I
  should check my VM.




• Other winners were for such things as the most
  entertaining SMS


• Large (Ultra 20 and 24" monitors) prizes for best blog
  and such




• Not that I am bitter but I would rather have the Ultra
  20 than the $200






• Much bemoaned by me, connectivity was a major issue




• Although apparently it might have been problems with my
  laptop
  almost as much as anything else, at least when the people sitting next
  to me were able to get access while I was not.




• Someone suggested brining the backend out to the front.




• I think that would have been great, as Dan
  indicated...this is a technical conference after all


• We should expose the technical aspects

Suggestions:

• LISA (USENIX) like content tracks 




• Refereed papers: A collection of 3 or so papers in a
  similar vein for an hour each


• Longer Presentation Slots (3 hours or so)






• WIPs: Short topics presentation overviews, here is what
  we are doing




• More accessable BOFs




• Tables at lunch was a good idea


• real BOF session availability would have been appreciated


• Too busy trying to get food and eat too really spend time
  finding all the groups




• Possibly longer, too much interesting information for the
  available time




• Maybe it is just me, I would have attended maybe 75% of
  the presentations given time




• A better scheduler




• 355 possible sessions (including duplicates)




• Presented in a list (depending on search criteria)








• Pick by topic/presentation (Content Catalog) would have
  been more useful




• Then given the list of desired presentations work on
  resolving conflicting selections






• Allow alternate selections to be maintained, I didn't
  attend all of the presentations I selected




• A capability to display only otherwise un-selected
  presentations for open time slots would also have been very useful




• A better view into content




• This would prefereably be the content catalog mentioned
  above






• I ignored presentations because I didn't know what they
  were really about


• It turns out that I would have liked to attend more and
  might have attended differently given better information




• Although see above: not enough time...

CEC Posts in Review:

I was going to collect
links to categories and such or individual bloggers/entries...too much
work so instead

Technorati:
cec2006

We have an internal survey on this year's CEC, I should go do that. If
there is one thing I am generally not lacking, it is opinions.

CEC: In Review

CEC2006: Too much content not enough time

Next Year:

• Kyocera KR1
• It seems that the state of high density WiFi may be slightly less than carrier grade, at least for temporary third party installations
• The good camera
• Not that the Cannon Point and Shoot is bad, just that I found myself wanting the more easily adjustable manual settings of the D1x
• The Managed Operations CEC blog/multi media Tiger Team (MOCECBMMTT? OK I made that up)

Interactivity(Web2.0):

• It was interesting, and hey I apparently won $200 for the most useful sms messages. I have yet to be contacted about my prize, maybe I should check my VM.
• Other winners were for such things as the most entertaining SMS
• Large (Ultra 20 and 24" monitors) prizes for best blog and such
• Not that I am bitter but I would rather have the Ultra 20 than the $200
• Much bemoaned by me, connectivity was a major issue
• Although apparently it might have been problems with my laptop almost as much as anything else, at least when the people sitting next to me were able to get access while I was not.
• Someone suggested brining the backend out to the front.
• I think that would have been great, as Dan indicated...this is a technical conference after all
• We should expose the technical aspects

Suggestions:

• LISA (USENIX) like content tracks 
• Refereed papers: A collection of 3 or so papers in a similar vein for an hour each
• Longer Presentation Slots (3 hours or so)
• WIPs: Short topics presentation overviews, here is what we are doing
• More accessable BOFs
• Tables at lunch was a good idea
• real BOF session availability would have been appreciated
• Too busy trying to get food and eat too really spend time finding all the groups
• Possibly longer, too much interesting information for the available time
• Maybe it is just me, I would have attended maybe 75% of the presentations given time
• A better scheduler
• 355 possible sessions (including duplicates)
• Presented in a list (depending on search criteria)
• Pick by topic/presentation (Content Catalog) would have been more useful
• Then given the list of desired presentations work on resolving conflicting selections
• Allow alternate selections to be maintained, I didn't attend all of the presentations I selected
• A capability to display only otherwise un-selected presentations for open time slots would also have been very useful
• A better view into content
• This would prefereably be the content catalog mentioned above
• I ignored presentations because I didn't know what they were really about
• It turns out that I would have liked to attend more and might have attended differently given better information
• Although see above: not enough time...

CEC Posts in Review:

I was going to collect links to categories and such or individual bloggers/entries...too much work so instead
Technorati: cec2006

We have an internal survey on this year's CEC, I should go do that. If there is one thing I am generally not lacking, it is opinions.

CEC: Managing Systems at Grid Scale (Our Presentations)

Shawn Ferry (ME) and Brian Smith





In general our  attendance was a little bit lower than I had
been hoping for. I think we got in the neighborhood of 80 total...I
think I should be able to find out exactly somewhere.



Both instances went fairly well we had the required Managed Operations
content. In the future we need to go straight to an elevator pitch and
be done with it. I watched as people got bored hearing about how
managed operations does things. Certainly they were interested in the
more technical details CTA, encrypted transport and such. Unfortunately
in our general Managed Operations presentation the less technically
interesting details are more prevalent and this is a geek
conference.



Both presentations had a few Managed Operations staff in attendance for
solidarity. As well as a couple of new Managed Operations staff from
APAC and EMEA.



The first presentation had a slight timing problem, the 15 min max
planned managed operations specific content ran about 30min, in the end
I got
my content in with about a minute to spare and Brian got the last bits
in about 5 after the scheduled end.  



The second presentation went more to plan. I was able to go into
more depth on the technical aspects of the monitoring. For those who
attended the presentation. We send an alert.



The only real failure for the whole thing was the attempt for a demo in
the extra 5min before the scheduled end of the second presentation
during the question and answer period. I had maintained what looked
like good network connectivity until I tried to use it, at which point
it all went pear
shaped and I lost network access again.



In brief we covered monitoring methodology:

We have a top down and bottom up approach.



Top Down(A Holistic View):

We start measuring from a user experience, can you reach the web site.

Then a complex web site walking script/library(That I
wrote) performs a:

• login


• job submission


• waits for completion


• downloads and analyzes the results


• deletes the job


• logs out

At various points in the process if we don't get specific results
(Welcome to the Grid,Job X submitted, Job X Successful, Logout
Complete)
or the process is taking too long we send an alert.



Really in the Middle Here(Business Process Monitoring):

The from a Managed Operations Point of view what we want to know is
what went wrong.



A user can get an login failure for a number of reasons:

• External authentication service could not be functional(Sun
  Grid does not maintain direct authentication data, you must have a
  sunsolve/mysun/store.sun.com account...Go  href="http://www.sun.com/identity">Federated Identity
  ( href="http://www.sun.com/software/media/flash/demo_federation/index.html">Silly
  Flash IDM) , see also: href="http://blogs.sun.com/saragates" rel="co-worker">Sara
  Gates Sun's VP IDM)


• Your name may have been found on the href="http://www.sun.com/sales/its/export/DRPL/index.html">Denied
  Restricted Persons List


• You may have moved to Brazil (This is a presentation In
  joke, I was hoping for some Brazilians to jump up and down and cheer)
  or really you are coming from outside the US.

A job can fail for a number of reasons:

• Malformed Job


• Portal Submission Failure


• Grid Engine Failure

Some of these things we can see cause from the middle and some we need
a lower level view.

Bottom Up (A small sampling, we monitor a very wide range):

• System level KPIs(Memory, CPU)


• processes (sge_qmaster,sge_execd)


• services(SMF)


• logfiles(errors, failures)


• disk utilization


• network devices (firewalls, switches, routers)

Examples:

• If the qmaster is not running the job will never actually
  get scheduled.


• If there are no execds running (slots available) the job
  will never get executed(or really scheduled because the qmaster won't
  schedule it unless it should be able to run.


• If queues are in an Error state they cannot run jobs.


• If the file system is full it is impossible for a job to
  write it's output

Presentation Things I learned:

• People want to know why they can't use the grid from
  outside the US and
  don't quite understand how it is that Sun can have a gird and Sun
  employees can't play with it.




• We knew this was an issue, we could have just addressed
  it straight from the slides




• Few Sun employees (who attended our presentation have ever
  used the
  SunGrid, of course I know that a good number of Sun Employees have used
  the Grid it would seem that we just didn't get a lot of them.




• A demo would have been well received, but if we were
  working with a demo we could have talked for hours




• I still need to remember to breathe fully. I find that I
  will talk but
  forget to inhale full breaths if I am not paying attention.




• I just need to pay attention to my breathing. A pause to
  take a breath is ok...also it lets people absorb your output

CEC: RBAC Demystified

Brian Bianquart and Darren Moffat

style="font-weight: bold;">Role Based Access Control

What is a Role: An account on the system

Cannot directly login


Could be root (or any user)

What is a Privilege: An attribute of a process

Checked by Kernel

Authorization: given to users directly or through profile

...Cutting back on
following/outlineing until I see
something that I am less sure is readily available online and in docs...

One exec_attr table can be
used across Solaris 8 and 9,
Trusted Solaris (8) and Solaris 10

Here we have a
graphic I have never seen
before...took a picture but it will probably be lame.


I think maybe hand drawings scanned and added to the slides.

href="http://www.flickr.com/photos/shawnferry/261252689/"
title="Photo Sharing"> style="border: 0px solid ; width: 800px; height: 600px;"
src="http://static.flickr.com/113/261252689_892fac0221_o.jpg"
alt="A picture of an RBAC slide">

style="font-weight: bold;">Q:
Can we make it such that user and role profiles can be modified while
the user is logged in or the role is in use.


A: Yes, that
is a bug fixed in update 3...changes may not take effect until next
login, but you will be able to make the change.

Standard RBAC
example:

Execute with elevated
privileges...Start Apache as a regular user on port 80


(As opposed to start as root and drop privs)

I think I was
hoping for more in depth technical details, still time yet we will see

/usr/bin/pfexec is the
closest thing to sudo only without authentication (yet)

pfexec will use the first
profile found....that is the ALL role should be last, otherwise don't
bother to define other profiles.

SMF demo: Allow a user to
change the running state of a service but not the boot state


e.g.

ALLOWED: svcadm enable/disable -t

DISALOWED: svcadm enable/disable (no -t)

DO NOT MODIFY SYSTEM
SUPPLIED PROFILES


File a bug if you think it should be changed


OR


Create your own profiles

Privileges

Kernel no longer only
checks for UID==0


48+ privileges checked instead

Now privilege
sets, next how the privileges flow not really going to note that
down...I know it is well documented I have read it.

Note: Dark Red on
black...hard to see, shouldn't do colors that evaluate to black

Use ppriv -D to debug
privilege access. (Yes this is commonly known)

ACLs

Solaris 10 NFSv4/ZFS ACLs
now match those as implemented in Windows NT/XP=

More info 

There is a RBAC and SUDO
comparison slide

Strengths and weaknesses
on both sides the most common requested deltas are being addressed.

Authentication and Netgroups are on/near the top of the list.

security-discuss@opensloaris.org
 and Sun blueprints

CEC: Dtrace Approaches to Real-World Problems

Jim Fiori

Apparently the presentation
is normally a 3+ hour presentation and Jim requested 3 hours, but all
the slots are 60min.

This is a presentation on approach not learning D.


This entry is a horrible hack job on content, too much and I am
learning more than I


can easily condense/digest usefully on the fly.

INTRO

Everyone needs
dtrace...but it is not the first thing to run.


Identify a possible issue use Dtrace to figure out what is going on.


Advice: Practice, Practice, Practice

Approach

use the manual all
examples are in /usr/demo/dtrace


use quantize(), min()/max()/avg() can hide data




Be careful using the PID provider it can impose load on a highly active
process.




Normal system tools still have their
place.

New tools: intrstat (some
others I haven't used)

Privs -- Root level or RBAC...

Zero Probe Effect -- via
instruction replacement

Scenarios

High User Time

hotuser.sh dtrace toolkit

C++ Apps

Watch for small
allocations and short allocations

High System Calls
(>100s)

use aggregation

use pfiles to determine target of File descriptor

System time (>10%
or user:sys near 2:1)

prsatst to
find it, dtrace to examine it

Threaded App.

prstat to find it


plockstat ... to see it, single process

Java

use jstack

Java 1.6 has static dtrace providers

Oracle

Look at I/O and File
systems first


ONLY after regular investigation by DBA (statspack etc)

Sybase

Watch for TCP Nagle
(buffering requests before sending)


Try TCP no delay on client and server

File system

Watch for
periodic pauses check autoup in large memory (>8G) systems

Hints and Tools

• Use a sample rate at not quite 1000, to help keep the Dtrace
  probe func from running with kernel actions 
  
  


• Dtrace is running in kernel context...you can't just dump
  the memory location, you need to dump the user memory space
  
  
  


• .mul() and .div() are SPARC V7...the application should be
  recompiled
  
  
  


• -c flag...use pid$target ... woot
  
  
  


• look for system call errors, when things are not working as
  expected
  
  
  


• see atomic_ops(3c) on mutex_lock/unlock for simple variable
  updates
  
  
  


• High ctx...look at FX scheduler (good for some apps)
  
  
  


• High Migrations...look at binding to a specific processor
  
  
  


• libumem for memory leaks
  
  
  


• Chime (GUI...uses Sparklines!!!...I like sparklines)
  OpenSolaris
  

Coming

• RFE 6311947 svn_43...fun/mod/ufunc/umod
  
  
  


• Dtrace in a zone
  
  
  


• aggregation printing
  
  
  

CEC: Photos

href="http://www.flickr.com/photos/18629766@N00/256792653/in/set-72157594306346526/" title="Cec2006" class="thumb_link" id="set_thumb_link_256792653"> src="http://static.flickr.com/87/256792653_c7f84340d9_s.jpg" alt="Cec2006" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/256727195/in/set-72157594306346526/" title="cec cameraphone test post" class="thumb_link" id="set_thumb_link_256727195"> src="http://static.flickr.com/104/256727195_4986836887_s.jpg" alt="cec cameraphone test post" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059897/in/set-72157594306346526/" title="Plastic Surgery?" class="thumb_link" id="set_thumb_link_260059897"> src="http://static.flickr.com/114/260059897_38a1148379_s.jpg" alt="Plastic Surgery?" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059989/in/set-72157594306346526/" title="Central Comupter" class="thumb_link" id="set_thumb_link_260059989"> src="http://static.flickr.com/117/260059989_604c69426a_s.jpg" alt="Central Comupter" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059977/in/set-72157594306346526/" title="A warehouse fire" class="thumb_link" id="set_thumb_link_260059977"> src="http://static.flickr.com/91/260059977_32200afea6_s.jpg" alt="A warehouse fire" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059960/in/set-72157594306346526/" title="A short timeout" class="thumb_link" id="set_thumb_link_260059960"> src="http://static.flickr.com/116/260059960_86b310f2f7_s.jpg" alt="A short timeout" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059945/in/set-72157594306346526/" title="A hectic breakfast" class="thumb_link" id="set_thumb_link_260059945"> src="http://static.flickr.com/81/260059945_7da1a2688c_s.jpg" alt="A hectic breakfast" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059935/in/set-72157594306346526/" title="PGE rebate" class="thumb_link" id="set_thumb_link_260059935"> src="http://static.flickr.com/109/260059935_8564035348_s.jpg" alt="PGE rebate" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059924/in/set-72157594306346526/" title="CEC Main stage" class="thumb_link" id="set_thumb_link_260059924"> src="http://static.flickr.com/118/260059924_1dc6d4670e_s.jpg" alt="CEC Main stage" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059911/in/set-72157594306346526/" title="The CEC tattoo" class="thumb_link" id="set_thumb_link_260059911"> src="http://static.flickr.com/89/260059911_72b6c37285_s.jpg" alt="The CEC tattoo" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059882/in/set-72157594306346526/" title="The view from my hotel window" class="thumb_link" id="set_thumb_link_260059882"> src="http://static.flickr.com/88/260059882_3a0cc9faa7_s.jpg" alt="The view from my hotel window" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059857/in/set-72157594306346526/" title="Foley's Irish House" class="thumb_link" id="set_thumb_link_260059857"> src="http://static.flickr.com/96/260059857_c1ad3689ba_s.jpg" alt="Foley's Irish House" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059832/in/set-72157594306346526/" title="Trouble Done?" class="thumb_link" id="set_thumb_link_260059832"> src="http://static.flickr.com/113/260059832_a01b025144_s.jpg" alt="Trouble Done?" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059808/in/set-72157594306346526/" title="Trouble Fulfilled" class="thumb_link" id="set_thumb_link_260059808"> src="http://static.flickr.com/118/260059808_60033ccab2_s.jpg" alt="Trouble Fulfilled" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059786/in/set-72157594306346526/" title="Trouble in action" class="thumb_link" id="set_thumb_link_260059786"> src="http://static.flickr.com/82/260059786_aaaa29096b_s.jpg" alt="Trouble in action" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059766/in/set-72157594306346526/" title="Trouble without a doubt" class="thumb_link" id="set_thumb_link_260059766"> src="http://static.flickr.com/93/260059766_3f052eb432_s.jpg" alt="Trouble without a doubt" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059741/in/set-72157594306346526/" title="A sure sign of trouble" class="thumb_link" id="set_thumb_link_260059741"> src="http://static.flickr.com/106/260059741_e6eb0c6b7f_s.jpg" alt="A sure sign of trouble" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059731/in/set-72157594306346526/" title="Stacia and Kevin" class="thumb_link" id="set_thumb_link_260059731"> src="http://static.flickr.com/99/260059731_2587ee4f2c_s.jpg" alt="Stacia and Kevin" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/260059717/in/set-72157594306346526/" title="Stacia, Mike and Doug" class="thumb_link" id="set_thumb_link_260059717"> src="http://static.flickr.com/104/260059717_d9b63e169a_s.jpg" alt="Stacia, Mike and Doug" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/259890294/in/set-72157594306346526/" title="A shoe on the street" class="thumb_link" id="set_thumb_link_259890294"> src="http://static.flickr.com/113/259890294_27438722ec_s.jpg" alt="A shoe on the street" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/258100082/in/set-72157594306346526/" title="Americas State of the Union" class="thumb_link" id="set_thumb_link_258100082"> src="http://static.flickr.com/92/258100082_0db6dd095f_s.jpg" alt="Americas State of the Union" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/257867375/in/set-72157594306346526/" title="Movies on a plane" class="thumb_link" id="set_thumb_link_257867375"> src="http://static.flickr.com/118/257867375_ea6b9d9c2c_s.jpg" alt="Movies on a plane" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/257465210/in/set-72157594306346526/" title="A full plane" class="thumb_link" id="set_thumb_link_257465210"> src="http://static.flickr.com/91/257465210_cac012e3d6_s.jpg" alt="A full plane" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/257457734/in/set-72157594306346526/" title="Doug in the isle" class="thumb_link" id="set_thumb_link_257457734"> src="http://static.flickr.com/107/257457734_c88ff17507_s.jpg" alt="Doug in the isle" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261253014/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261253014"> src="http://static.flickr.com/88/261253014_49e1e4d7da_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261253009/in/set-72157594306346526/" title="After Foley's Closed" class="thumb_link" id="set_thumb_link_261253009"> src="http://static.flickr.com/100/261253009_e59b41dc1a_s.jpg" alt="After Foley's Closed" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261253006/in/set-72157594306346526/" title="Ian White?" class="thumb_link" id="set_thumb_link_261253006"> src="http://static.flickr.com/91/261253006_2321c73313_s.jpg" alt="Ian White?" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261253002/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261253002"> src="http://static.flickr.com/92/261253002_afe9ea58d6_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252993/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252993"> src="http://static.flickr.com/96/261252993_af77e363d3_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75">  href="http://www.flickr.com/photos/18629766@N00/261252985/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252985"> src="http://static.flickr.com/120/261252985_8a6d784af1_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252979/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252979"> src="http://static.flickr.com/117/261252979_e63b76a523_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252969/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252969"> src="http://static.flickr.com/120/261252969_ed4bb22ca3_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252961/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252961"> src="http://static.flickr.com/97/261252961_cee5b87310_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252950/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252950"> src="http://static.flickr.com/88/261252950_dfd1cb88f9_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252940/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252940"> src="http://static.flickr.com/110/261252940_72d96bcf7e_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252934/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252934"> src="http://static.flickr.com/96/261252934_fc8afaae6c_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252925/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252925"> src="http://static.flickr.com/107/261252925_db3c95361e_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252923/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252923"> src="http://static.flickr.com/96/261252923_467bc6ae5e_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252914/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252914"> src="http://static.flickr.com/112/261252914_7bac894fe4_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252903/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252903"> src="http://static.flickr.com/80/261252903_33ccd6628d_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252900/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252900"> src="http://static.flickr.com/80/261252900_5230f8cc73_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252894/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252894"> src="http://static.flickr.com/120/261252894_db55a083bc_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252889/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252889"> src="http://static.flickr.com/89/261252889_54d02bba12_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252884/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252884"> src="http://static.flickr.com/106/261252884_1282be2f0d_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252878/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252878"> src="http://static.flickr.com/92/261252878_4d02610c61_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252869/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252869"> src="http://static.flickr.com/120/261252869_d933d3aa88_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252859/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252859"> src="http://static.flickr.com/102/261252859_860666d91e_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252852/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252852"> src="http://static.flickr.com/102/261252852_49460fa77b_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252843/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252843"> src="http://static.flickr.com/107/261252843_ec883f081c_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252835/in/set-72157594306346526/" title="Foley's Irish House The Party Keeps Going" class="thumb_link" id="set_thumb_link_261252835"> src="http://static.flickr.com/100/261252835_1941df9cd7_s.jpg" alt="Foley's Irish House The Party Keeps Going" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252828/in/set-72157594306346526/" title="Managed Operations" class="thumb_link" id="set_thumb_link_261252828"> src="http://static.flickr.com/84/261252828_813c288ce3_s.jpg" alt="Managed Operations" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252822/in/set-72157594306346526/" title="Space Invaders" class="thumb_link" id="set_thumb_link_261252822"> src="http://static.flickr.com/88/261252822_f9666ad08f_s.jpg" alt="Space Invaders" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252817/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252817"> src="http://static.flickr.com/94/261252817_4075c28fa9_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252814/in/set-72157594306346526/" title="Prize Raffle" class="thumb_link" id="set_thumb_link_261252814"> src="http://static.flickr.com/117/261252814_088c79ab3f_s.jpg" alt="Prize Raffle" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252811/in/set-72157594306346526/" title="Miss. Hula Hoop" class="thumb_link" id="set_thumb_link_261252811"> src="http://static.flickr.com/120/261252811_13911649f7_s.jpg" alt="Miss. Hula Hoop" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252801/in/set-72157594306346526/" title="Miss. Hula Hoop" class="thumb_link" id="set_thumb_link_261252801"> src="http://static.flickr.com/111/261252801_42b3ddbd7e_s.jpg" alt="Miss. Hula Hoop" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252798/in/set-72157594306346526/" title="Miss. Hula-Hoop" class="thumb_link" id="set_thumb_link_261252798"> src="http://static.flickr.com/104/261252798_8003d2b5a0_s.jpg" alt="Miss. Hula-Hoop" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252793/in/set-72157594306346526/" title="Acrobatics and Shadows" class="thumb_link" id="set_thumb_link_261252793"> src="http://static.flickr.com/109/261252793_1d8a1fab9f_s.jpg" alt="Acrobatics and Shadows" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252788/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252788"> src="http://static.flickr.com/122/261252788_5b63192e1b_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252783/in/set-72157594306346526/" title="Acrobatics and Shadows" class="thumb_link" id="set_thumb_link_261252783"> src="http://static.flickr.com/103/261252783_44ddfa889e_s.jpg" alt="Acrobatics and Shadows" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252777/in/set-72157594306346526/" title="Feats of Strength and Balance" class="thumb_link" id="set_thumb_link_261252777"> src="http://static.flickr.com/113/261252777_5624ab3eb3_s.jpg" alt="Feats of Strength and Balance" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252772/in/set-72157594306346526/" title="Feats of Strength and Balance" class="thumb_link" id="set_thumb_link_261252772"> src="http://static.flickr.com/57/261252772_594b48f8a3_s.jpg" alt="Feats of Strength and Balance" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252768/in/set-72157594306346526/" title="Feats of Strength and Balance" class="thumb_link" id="set_thumb_link_261252768"> src="http://static.flickr.com/95/261252768_9fb79d188b_s.jpg" alt="Feats of Strength and Balance" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252760/in/set-72157594306346526/" title="Feats of Strength and Balance" class="thumb_link" id="set_thumb_link_261252760"> src="http://static.flickr.com/91/261252760_6672535ea0_s.jpg" alt="Feats of Strength and Balance" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252754/in/set-72157594306346526/" title="CEC Attendee PArty" class="thumb_link" id="set_thumb_link_261252754"> src="http://static.flickr.com/105/261252754_698f5478d4_s.jpg" alt="CEC Attendee PArty" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252747/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252747"> src="http://static.flickr.com/117/261252747_e435f4e251_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252741/in/set-72157594306346526/" title="A view of the second floor" class="thumb_link" id="set_thumb_link_261252741"> src="http://static.flickr.com/109/261252741_7a13d265bb_s.jpg" alt="A view of the second floor" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252735/in/set-72157594306346526/" title="IMG_2416.jpg" class="thumb_link" id="set_thumb_link_261252735"> src="http://static.flickr.com/86/261252735_8ec8ab4e2d_s.jpg" alt="IMG_2416.jpg" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252729/in/set-72157594306346526/" title="IMG_2415.jpg" class="thumb_link" id="set_thumb_link_261252729"> src="http://static.flickr.com/102/261252729_5f40a75b7e_s.jpg" alt="IMG_2415.jpg" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252723/in/set-72157594306346526/" title="IMG_2414.jpg" class="thumb_link" id="set_thumb_link_261252723"> src="http://static.flickr.com/91/261252723_b6f73a540d_s.jpg" alt="IMG_2414.jpg" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252717/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252717"> src="http://static.flickr.com/118/261252717_8b26087863_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252712/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252712"> src="http://static.flickr.com/84/261252712_38fac956ec_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252708/in/set-72157594306346526/" title="CEC Attendee Party" class="thumb_link" id="set_thumb_link_261252708"> src="http://static.flickr.com/87/261252708_7e280c685c_s.jpg" alt="CEC Attendee Party" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252701/in/set-72157594306346526/" title="Nerds Surrounded by Twinkies" class="thumb_link" id="set_thumb_link_261252701"> src="http://static.flickr.com/102/261252701_128bb8a026_s.jpg" alt="Nerds Surrounded by Twinkies" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252689/in/set-72157594306346526/" title="A picture of an RBAC slide" class="thumb_link" id="set_thumb_link_261252689"> src="http://static.flickr.com/113/261252689_892fac0221_s.jpg" alt="A picture of an RBAC slide" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252684/in/set-72157594306346526/" title="My view in the morning" class="thumb_link" id="set_thumb_link_261252684"> src="http://static.flickr.com/100/261252684_519adb42c6_s.jpg" alt="My view in the morning" height="75" width="75"> href="http://www.flickr.com/photos/18629766@N00/261252678/in/set-72157594306346526/" title="I woke up typing" class="thumb_link" id="set_thumb_link_261252678"> src="http://static.flickr.com/84/261252678_4a5bbba3f4_s.jpg" alt="I woke up typing" height="75" width="75">