mnn

fork: Resource temporarily unavailable

Jacob Redding — Tue, 28 Aug 2007 19:46:48 +0000

This afternoon I took the time to upgrade the userpoints module to version 3 merging in point expirations, point categorizations and an API change. Most of this work was already completed but was sitting on MNN's dev server. When I logged into MNN's server I was greeted by a rather unfriendly shell message of " fork: Resource temporarily unavailable". Although I didn't think much of it the minute my tune quickly changed as I was also given that message when attempting to vi my bits of code.

oh boy.. this is going to be fun

Having no idea what this error was truly referring to I immediately tried to get a process list to see what was running on the server. Again I was greeted by the same error. I then jumped over to /var/log to check the syslog.

The syslog was zero bytes!!! NOT GOOD

Now it turned from a minor inconvenience to a serious issue. The syslog file was hosed, I couldn't get a process list, in fact I couldn't do anything. Thoughts of hacking, being owned, etc. flew through my mind and also through Andy's, whom I had quickly contacted on iChat to help resolve the issue (after all its his box now).

I tried to su to root to elevate my authority. No luck. Not a password problem though no resources to spawn the shell.

Andy rushed over to the server room to try the direct terminal, same issue (which made sense).

Google to the rescue! Apparently we had a resource issue. We needed resources on the box before we would do anything. A quick reboot would have solved the issue but it would have also destroyed the evidence of whatever was causing the issue. The next step was to start killing off processes. but how?

ps wouldn't run due to lack of resources and I couldn't cat/more/less anything in /var/run. The init.d scripts would run due to lack of resources. Heck ls wouldn't even run. The only application I could run was "kill" but not "killall". What a wonderful predicament. I have a method to kill processes but no way to know of a process to kill.. or did I?....

It turns out that the most helpful feature in bash was actually the most helpful. <tab> <tab> would return a directory listing if coupled with cd. Even though I couldn't use ls I could do a cd <tab> <tab> to get the directory listing. The first thing I did was a cd /proc/ <tab> <tab>. 8,300 items!!! This is a development box it shouldn't have more than 150 items. Some daemon or application must have gone crazy and spawned many many child processes. But which one?

If I had cat I could look in the cmdline "file" of the /proc/<id>/cmdline to determine what was what but without resources the only thing I could do was "kill" and even that only worked every other time.

My solution, although not perfect, was: Kill off the highest numbered processes as they will, most likely, be related to runaway daemon/application. Again bash to the rescue! This quick script helped to save the day

for i in /proc/3????
do
sudo kill -9 ${i##*/}
done

Fortunately I was in the sudo file for the kill application so I had the authority to kill without remorse (-9). The script didn't complete on the first run, it ran out of resources but it DID kill processes. I kept running it and after about 3 runs the system start to breathe.. a little. I was eventually able to properly shut down apache2 with the init script (i.e. /etc/init.d/apache2 stop. This freed up enough resources for me to finally grab a ps listing.

RETROCLIENT from RetroSpect!!!! was all over the place like Jackson Pollack Linux edition

I quickly killed the parent process of retroclient which took all of the children with it. The listing of /proc immediately dropped from a listing of over 8,000 to 113. Oh Retrospect how do I love thee? Let me count the ways....0

Talk Radio, MNN and funny business

Jacob Redding — Fri, 01 Jun 2007 04:16:23 +0000

This evening Diana and I went to see the Broadway show Talk Radio. In a word the show is Amazing in two words the show is "F*in Amazing" in three words it is "Totally F*in Amazing" and in California speak it is "dude, like totally awesome n' such". I could ramble on and on about the cast and how brilliantly they acted but I won't; that's boring, go read an actual review.

I will, however, say that it is a brilliantly written play that kept me on the edge of my seat the entire time despite the big spikey haired head seated in front of me. No really, there was no body attached to the head it was just one very large head.

The reality of the show is that it is a single set play set in a 1980's radio station and you are watching a 2 hour long "Shock jock" style talk radio show. Diana more elequently stated that it was "watching a man confront his inner demon". Either way it was good but there was one part in the play that related to me in a more "hey, that's me right now" type manner.

The "station manager", Dan, came out in the middle of the show and did a monologue about his relationship to "Barry Champlain", the talk show host and star of the play (depicted in the picture to the left).

Dan came out and stated how he had found "Barry Champlain" and molded him into what he needed to create the perfect radio station. Now that piece didn't have any relationship to my current situation but his next statements did. He created an analogy between radio hosts and trains, most of which made no sense.

One sentence, sadly, made sense "The difference between Barry and I is that I know this is only a job and I can leave it at the door"

Ok so if you're reading this and you expected a review of Talk Radio you're about to be thrown into my immediate world. After all this is my blog about my life. So sorry I won't be offended if you leave now. Go see the show, its good.

My current situation is exactly that. I've been working at MNN for the past 19 or so months and I took the job not as a job. I had just left a very corporate job as an accountant at Intel and I wanted to get back to something I love doing (IT, geeky work) and I also wanted to actually do something. Upon arriving in New York I relaxed for a few weeks. Then I started looking for a job (aah savings. gotta love it). I only applied at places that looked interesting and were working for/with/towards causes that I cared about. I was also ONLY looking at non-profits and places with a social-cause.

I applied at a handful of places and had a number of interviews but the ONLY place that looked remotely interesting was MNN and I saw a ton of potential with the organization. My initial assessment of the place was that they were in need of a decent IT guy but nothing serious (they only have a staff of 50 after all). I also noted, to myself, that the place would be extremely flexible to really explore and "flex", so to speak, my IT/Finance/managerial skills. After all I'm a geek with a Masters in Finance, I have to be able to do something with that right?

After I took the job I realized that there were some very serious problems at MNN. I took the job head-on though and hunkered down for the long road ahead. I worked too many 16-20 hour days to count and I kicked some serious ass at MNN. Yes, yes I'm being extremely egotistical and arrogant but I think I can back it up. Not only did I rebuild a physical and data infrastructure at MNN but I also worked on building a community and infrastructure to support MNN long after I'm gone. I attended multiple conferences around the country to mobilize access centers together and got heavily involved in the Drupal community in NYC (to the point where I starting hosting Drupal Camps and the monthly meetups).

In my opinion building a supporting infrastructure is more important that fixing the problems of today. My entire vision, and thus most of work, has always been focused on the 5, 10 and 15 years from now. Positioning MNN so that they are prepared to handle the hurdles the Internet is going to throw at Public Access (and cable in general) was my focus.

Ok so now back to the point. In accepting my position at MNN I took the very first salary offer they threw at me without negotiation. I have never asked for a raise and I have had a low impact on MNN's budget; to the extent of working on a laptop without a working screen for almost a year (seriously). I did it because it wasn't a "job" to me, I was building something, I was helping MNN, and public access, to position themselves for the future.

Well, sadly, it bit me right in the ass.

The poor managerial infrastructure at MNN became too much for me handle. After fighting against a lack of standard procedures, no budgets, no financial reports, an inability to hire people and an increasing list of problems, that have been solved in most businesses, I finally had enough. Months ago I tendered my resignation which would have been effective in August. Later, after a rather emotional managerial meeting that threw me over the edge, I stepped up my resignation to June 1st.

My resignation relieved a lot of my daily stress and the reality had set in that most of what I had done at MNN was done and my standard day had become filled with the 9 to 5 tasks. I say done, however, reluctantly because one project wasn't done.

Over a year ago I made a very important decision at MNN. Now truth be told the decision wasn't solely me. MNN has a Board of Directors, an Executive Director and an entire chain of command so I made a recommendation that was followed by the "powers that be" but when it comes down to it, I made the decision. I decided that an application developed by an unnamed vendor was developed poorly. This application cost MNN approximately $110,000 and I recommended scrapping it, effectively throwing away all the work (and thus cash)

I stand by that decision.

I also made the decision to spend another $60,000 on a redevelopment project.

A year into a 3 month long project (i.e. 9 months overdue) the project still isn't completed and I have resigned from the organization. I, however, made an extremely important decision and I want to see it through to fruition (oh the fruits will be juicy too). Despite my incredible frustration with the organization I offered to continue as an employee to work as a project manager to complete this project. I did/do not want to see MNN get hurt again by another failed software development project.

I want this to be clear. I do not have to do this but I made a decision and I'm sticking by it. Additionally I'm sticking by it at my standard salaried employee rate (which is/was already low) and I am by no means "setting up a sweet deal for myself".

Unfortunately MNN hired a new business manager about 6 months ago. A really nice woman that is extremely talented, smart and highly educated. A CPA that has worked with numerous corporate and non-profit clients. I was enamored by her intellect and dedication during her interview and I was one, of many, that immediately said "Hire her. She's the one. Hire her. She will do great things here". Truth is she is going to do wonderful things at MNN but unfortunately in the process of doing those wonderful things she is going to cause a lot of tension and some decisions aren't going to be the "right" ones. Par for the course, I guess.

back to me and the Talk Radio discussion. I didn't treat MNN as a "job" and that was my mistake.

Despite offering to continue working as an f/t employee of MNN she decided that I needed to leave MNN and be rehired as an outside consultant complete with a contract.

Sadly this means that I have to pay my own Employer/Employee taxes, pay for my own benefits, accrue my own vacation, etc. Not to mention having to develop contracts, go through fee negotiations and losing my standard employee rights afforded under standard federal, state and city laws.

Although this is a standard procedure in the business world it is only standard procedure because of abuse and business manipulating accounting rules. I tried to play nice and break through these so-called "corporate" barriers in the U.S business world. Items such as lengthy contracts and untrusting fee negotiations where waste is commonplace.

Despite putting my heart and soul into this organization I have been treated as an "employee", a replaceable, temporary untrustworthy "person" that must have contracts, legal waivers, distinction and precedence between myself and the "corporation".

I can honestly say that I tried to have a minimal impact on the organization while providing maximum value back and I was bitten hard. Extremely hard.

Goodbye MNN, Hello China

Jacob Redding — Tue, 01 May 2007 14:22:31 +0000

I announced my resignation from MNN today sending a short to-the-point note to entire staff. I told key staff, including the E.D, of my resignation weeks, and in some cases months, before finally settling upon a date.

My last day at MNN will be June 1st.

There were a lot of factors leading to my decision to leave MNN. The primary reason is that Diana and I have decided to move to China. Yes that's right we are moving to China. The exact dates of departure and the arrival cities are still unknown but those are details we can fill in later.

The other factors are more short term superficial items that MNN can solve rather easily. Burn out played a major role in my decision. MNN is going through a lot of transitions and because of these transitions I found myself putting in more 15+ hour days than I care to remember. I have slept at the office on more than one occassions due to the sheer amount of work that needed to be done. Of course this doesn't take into account the out of work networking, conferences and personal time dedicated to cheerleading/advancing MNN in some aspect. In the 18 months I worked for the organization I was able to do a very large amount of work spread across many disciplines.

A quick list includes

Redesigned/Implemented a new physical and IP network
Installed configured

LDAP services
email services (including postfix, imapd, mailman, etc.)
DHCP, DNS, WINS, AFP, Cifs services
Internal and External firewalls (iptables)
Webserver
Tape backup system (including offsite storage and disaster recovery plans)

Developed plan and found vendors to install IP Telephony system (reducing monthly costs)
Wrote several hundred pages of documentation
Reworked many vendor contracts and established support contracts
Installed a 10mb IP fiber line w/o raising costs; providing significantly improved service to the organization
Setup employee/producer discount Apple purchases
Created a system that can easily handle p/t, interns and volunteers without compromising security or significantly increasing management support
Redeveloped MNN.org and setup translation system.

Of course I didn't do all of these things solely on my own. I was responsible for hiring individuals to help me accomplish all of this. It is also those individuals and positions that are remaining at MNN to help sustain this in the future. In my opinion this was a large part of my position. I was not simply building something at MNN but I was developing a sustainable system for MNN.

I honestly feel as though I am leaving the organization in a significantly better position that when I took the position. If I were to stay another 3 to 5 years I could do even more, however, I have reached a point of where it is time for someone else to take the reigns. A new administrator can take over the network without having to build much and can take a much more relaxed sustaining role. This person can focus on providing better staff support and planning for the future.

Before moving to China and, most likely, while in China I am shifting my job role to Drupal development. There is a significant amount of work being done in Drupal that focuses on the non-profit sector; including software used by MNN. I think that I can add some value in this industry, helping to plan out and develop software that can be used by non-profits nation (world?) wide. While I could have done this work at MNN leaving MNN will allow me to better focus on the development.

I will miss working at MNN but I'm incredibly excited about moving to China.

Last day of work at MNN

Jacob Redding — Mon, 30 Apr 2007 05:29:16 +0000

It is with great sadness that I announce that this will be my last day at MNN. I worked for the computer for close to two years and the time has come for me to move on.

The decision to leave MNN was not made lightly nor abruptly. A few months notice was given to both upper and lateral management and an amicable last day was set. Diana and I decided to leave New York for the "redder" pastures of China. I have been studying Mandarin on and off for almost a year now and it is time for me to get serious about learning the language.

MNN was an absolutely wonderful place to work and I hope that when we return to New York in a few years I'll be able to jump right back into MNN.

I will miss working at MNN.

Vegas for NAB

Jacob Redding — Fri, 20 Apr 2007 05:49:14 +0000

I'm off to Vegas for the NAB conference. MNN is rebuilding its Master Control and doing many other things. I'm heading to the conference to look at new equipment and to meet with several of our vendors.

I'm not a fan of Vegas but the conference is always interesting. This will be my second appearance at NAB.

O'Reilly, Harvard, Twitter and Vegas baby!

Jacob Redding — Mon, 16 Apr 2007 18:11:55 +0000

Check it out I was mentioned on an O'Reilly blog. I spoke briefly on using video w/ Drupal during DrupalCon (OS-CMS sunnyvale, Ca) and I get a one-liner on O'Reily. Man I should speak more..

Oh wait. I am. At Harvard.

The Beyond Broadcast Conference is occuring in Cambridge, Mass next week and I am on a panel speaking about "Participatory Media" and "Open Media publishing Tools". Basically it is geek talk for an entire day. I'm excited. I love this stuff. Check it out on Wikipedia

In other news I have fallen in love with twitter. Twitter is a stupidly basic service that allows someone to send message to the twitter service via the web, IM or SMS. So even when I'm not near a computer I can send simple messages to my blog to update my status/whereabouts with everyone. You can either read it on my webpage (the block on the Right), subscribe to an RSS feed, get the updates on your phone, or see them on twitter.

If you read my twitter messages from this week you would have noticed that I went to Las Vegas. I flew in for the NAB conference, which is a massive conference for television stations around the world. Tens of thousands of people attend and thousands of vendors show off their products. People arrive from around the world, hang around for the week and talk "shop" with other television heads. I'm not fond of Vegas but the conference is actually quite interesting and fun at times.

Hyping Civicrm

Jacob Redding — Sat, 17 Mar 2007 01:08:10 +0000

I arrived in Denver almost a week ago and have been working 14-18 hours a day on MNN's database open source project. A massive project involving the creation of custom Drupal modules, integration with CiviCRM and tons of coordination.

Since arriving in Denver I have learned two things.
1) Deb (g/f of Tony) showed my hypem.com -schweet!
2) CiviCRM is a horrible system to program against

Oh well the struggle continues. In one week's time I will be on vacation in Portland, Oregon! I can not wait to get back to Portland.

waz you up to yo!

Jacob Redding — Thu, 26 Jan 2006 05:13:00 +0000

Ok so if you haven't realized by now I accepted a job at MNN and I have been completely bogged down by this stupidly insane project with a crazy software developer. At this point I'm not sure what is was the software developer that didn't even follow software 101 guidelines or the amount of bureaucracy created by a non-profit organization. At this point I am seriously considering finding a corporate job (half joking/half serious) because if I don't get out from under this current project I will never be able to do things that I wanted to do at MNN. My job there would be pointless and I much rather spend 60 hours a week making 30k more in a corporate hell hole of a job.

Anyhow enough with that rant. A few weeks ago I went on a Timesup New York memorial Bike ride. It was a slow ride that has 4 different starting points in each of the 4 outer borroughs (Brooklyn, Queens, Staten Island and the Bronx). The 4 groups of people then met up in Manhattan and rode around to various spots where bike commutters were killed while riding their bikes. At each of the 16 spots a memorial was placed and one spot in each borrough (5 total) an "Ghost bike" was placed to raise public awareness of bike commutting and the danger put on it by the carelessness of drivers.

It was a great ride albeit incredibly somber.

You can check it out at a few places.
Here is a Video
Pictures (I'm in one about half-way down the page)
Village Voice article on the event

and of course check out Times Up New York

Ride to work with me

Jacob Redding — Sat, 21 Jan 2006 06:16:00 +0000

I woke up this morning feeling great. Last night I stayed at work until the early morning and moved servers, installed software and made our database software, at MNN, ridicuously redundant and on overkill hardware. I don't want to have to mess with the software ever again.

I had a really good nights sleep and I woke up feeling good and refreshed. I grabbed my video camera and headed to work. I wanted to shoot a short (well 7 minutes short) video of my ride to work so that you (yes you) can see what I see everyday. All in all it wasn't so bad.

A lot of fun stuff in the video, you can also see the toll that work puts on me. Fortunately Monday is the FINAL day that I have to deal with this Ocean 7 software development firm. Schwweeet..

I'm very much looking forward to this weekend. Learning more Mandarin and just not thinking about work.

The Video is attached below.

Disputes, Lawyers and craziness

Jacob Redding — Tue, 13 Dec 2005 23:04:00 +0000

Oh man today was one of those days. A company that I have been working with, as a representative of MNN, called me today. They received a letter from our lawyers expressing our frustration with a software product that they developed. Unfortunately it was not a fun phone call and if I end up dead tomorrow you know where to start looking; Brooklyn, no wait I mean I live in Queens, look for me in Queens---ya that's it--- Queens.

Anyhow I can't say anything about it but it was not a fun decision to make but I had no choice.

uggghhh, you have to love business.