Skip to content

mandate of action for climate change by Kofi Annan

Wednesday, September 23, 2009

One unique opportunity available as a student at Columbia is the frequent visitation of world leaders, particularly around the time that the UN general assembly meets. A visitation this year was by Kofi Annan, former secretary-general of the UN. Although terse, these notes are a brief sketch of what he discussed at Columbia. Two interesting differences from previous visiting guests was that (1) his talk was tailored in many places to speak to the community of students and faculty members here and (2) in an effort to incite activism, he came with his own rock video to be released for download on October first as a new-media form of political petition. As only a passive participant in blogging and the online social community, I can only wish that more people were interested in at least learning more about this movement instead of the latest hollywood flub.

Annan Summary Annan Introduction
Speech snippit…

Gabriel Silver Memorial Lecture with Kofi Annan

background

  • Topic: global climate change
    • Wednesday, September 23, 2009, 12:00 pm, Rotunda, Low Memorial Library
  • acceptance of position as a global fellow at Columbia
    • rooted from one of the first members, Eisenhower, appointed before serving as president
    • spoke towards equality and freedom in each country
    • sense of global community, achieved at Columbia is one that is addressed today
  • climate change is not the only threat: conflict, famine, disease, poverty — each will worsen with climate change
    • famine will increase as water resources are used, rise of sea levels, and the acceleration of disease
    • more work is needed, not only through humanitarian reports, but also direct change to avoid some impacts that are at a much faster pace than predicted
    • report issued this summer documented impact on thousands in poorest nations

political change is required

  • urge leaders of g20 to catalyze movement; they are responsible for the majority of greenhouse emissions
    • agreement about impact of gasses thus far has not lead to needed political momentum
    • post 2012 agreement must be poignant and binding
    • those that are most threatened live in poor countries in asia, africa, or small island states but have contributed less than 2% of the emissions — current requirements cut by 19% by 2050
    • more aggressive and needed goal is to cut by at least 25% by 2040
  • other countries feel that economic opportuniteis must not be threatened – correct, but must still abide by reductions
    • fairness dictates that rules should apply to all (i.e. bicycles should not make up for SUV drivers)
  • least developed countries deserve incentives for any reduction that can provide through green land use, reforestation, etc
    • need additional funds as well as transfer of resources and low-carbon technologies

financial requirements

  • issues that still need to be resolved include cost of adaptation and going forward — sums so far are no where near enough
    • cost of climate change in Africa will raise from 30 billion in 2013, rising to 50-100 Billion through 2020
  • funds may come from pricing carbon not only to provide funds but provide a real reason to reduce consumption and production
  • need leaders to put aside narrow national interests and focus on long-term well-being of future generations
    • US will need to be the leader (China has overtaken the country as a whole) but per-capita is 4x China and 2x Europe
    • Agrees that Obama needs to take change now, but can not if stopped at home by populous

leadership is required in many places

  • leadership is not confined to politics alone, must also come from business aspects
    • both in minimization and investment in technologies abroad; many advances here will help to combat changes in the future
  • the academic world can also lead by brining different disciplines together
    • not only bridging different disciplines, but also through social opportunities to utilize climate change opportunities
    • join the ‘tik tok global campaign for climate justice’ that raises awareness and fair debate in Copenhagen

problems to be overcome

  • striking comments by Eisenhower was his optimism
  • believe if community can come together, it can provide a new page for international cooperation and understanding
    • currently students are first generation to consider themselves ‘citizen of the world’

questions from the audience

  • regarding economy, africa must decrease birth rate to achieve economic development in conflict to Pope’s claim of decreasing birth rate
    • climate change is still making this more difficult (drought, food production , floods, spread of diseases)
    • certainly some countries (like Zimbabwe), the lifespan has dropped by more than 10 years, loosing farmers, doctor, and teacher indicating a loss of the present as well as the future
    • measure are being taken both by the government and foreign firms; example is ‘green revolution for africa’
  • what is the most effective way for students to raise awareness
    • network and work with friends in groups that work on climate change; set examples yourself
    • generally work to push issue higher in the political agenda
    • showing that progress has been made, science is no longer questioned
  • view of US as a leader in climate change
    • world reaction to Obama’s election was very favorable and positive; often considered it a part of the world’s election
    • this issue goes beyond Obama though because it indicates that US and industrialized countries must lead…
      • if largest polluters continue to do so, developing countries will feel it is also there right as they grow
  • thoughts on efforts in Africa for leaders to be more accountable to their people in terms of dollars given
    • see development of society that is trying to keep government responsible; impressed with civil societies that have developed, but more is needed
    • people are demanding transparency, fighting in committee, and want honest security forces — but this progress is also from students coming to the US and others that have gone back to affect change
  • Nigerian genocide impact — why did you not take that moment to stop the violence?
    • must understand that UN has several forms: the UN in general assembly (pass mandates, etc) and secretariat (implements mandates)
    • in the case of Rwanda, at this time, international community was withdrawing form Somalia… once US withdrew, other industrialized countries followed suit (removing the best trained and best militarized)
    • set the precedent that troops were not available; Clinton apologized, other governments apologized….
      • but when violence was discovered, a few soldiers were killed and other armies pulled out, largely following US examples
    • in this situation, the UN failed; not for just the secretary but the entire community — problem is that response is only made once the problem has erupted, specifically, there is no standing army
      • example is currently impact in Darfour, needed only 18 helicopters for 2 years, but none were given internationally
  • anything that university can do to help students get involved?
    • often asked ‘what should young people do to become good global citizens’ — answer is to organize and take it on, even small problems
    • at university level there are other students and professors to help, but initiative must often be taken by the individual
    • problem perhaps more true in developing countries is that after receiving an education, some people have no link to the needs of their society — perhaps vocational training schools would be a better answer
      • problem also with parents who prefer a degree over practical experience for a country
      • could also encourage this opportunity in a university by asking for a course

final addition — a video for climate change

Filed in Freeform, Lectures | Comments (0)

quick MacOS VNC and Subversion fixes

Thursday, May 21, 2009

Just a quick couple of notes in this post. Two problems sucked a few hours of time this week, so it seems reasonable that they get pushed somewhere more visible for those with the best of Google-fu… connecting to a Mac VNC server from Linux and a weird “authorization failed” bug/oddity in subversion.

Connecting to MacOS X VNC from Linux

VNC is pretty fun and useful for remote-connecting machines. One particular usage that has served its purpose is solving a “my computer is broken” call from parents or siblings, inspired by VNC over SSH and SSH port forwarding to avoid those pesky firewalls. Connecting as a client from a Mac, Chicken of the VNC seems to work with little/no fuss for every scenario I had tried. Sure, there are other packages out there, but why use a commercial one that requires installation if something else is open-source and works fine?

For some reason the “screen sharing” option in OSX seems to return some faulty version information or it just doesn’t do a good job of negotiated color depth, etc. The result is that if you try to connect to a machine running the OSX version of VNC named Apple Remote Desktop or Screen Sharing in 10.5.7 (and earlier), the session you open will immediately close. Running VNC Viewer Free Edition 4.1.1 from a Debian distro, this error presented itself. No option on a GUI (i.e. the Terminal Server Client GUI) would work, as it requires the command-line option below (mandating the use of 32bit color).

vncviewer --FullColor=on

Of course, after you make this change and get VNC to work on your linux client, you may realized that it takes a *lot* of bandwidth to get a fluid response, which for me just wasn’t worth it in the end.

Here’s the original thread that eventually shed light on the subject… VNC Thread

I have to start vncviewer with –FullColour=on, else it terminates without showing the display. I think this means it’s using 32 bpp, even though I’ve configured the Mac to use “thousands of colours” (i.e. 16 bpp) in its display settings dialog, so it is wasting half of its network bandwidth.

Subversion rejects check-in with “Authorization Failed”

Another second annoyance was the sudden rejection of SVN check-in actions. If you had a nice, healthy repository that permitted the check-in or update of your tree and suddenly you’re getting rejection notices you might have the same problem:

localhost:cuzero quinone$ svn ci localtest 
emz@'s password: 
svn: Commit failed (details follow):
svn: Authorization failed
svn: Your commit message was left in a temporary file:
svn:    '/private/tmp/temp1/svn-commit.3.tmp'

Although the error itself is of no help, it’s easy to confirm that you have this problem…

  • you’re using ssh+svn:// to access your repository
  • you can ssh into your server without problems
  • you can ‘checkout’ or ‘update’ your local copy without problems
  • when you try to commit you get this ambiguous error.

Fortunately, there is a one line fix. Just add the line below to your “/your/repo/dir/conf/svnserve.conf” file.

[general]
anon-access = write

Assuming you already have unix permissions correctly configured to restrict access to that repo directory, you shouldn’t need to have explicit users defined in your svnserve file. Thus, you can allow anonymous write access, which is granted with the above line. Here’s a Subversion fix post that was found after a few hours of digging. It is still puzzling as to why this line was suddenly needed and all but the last possibility has been ruled out: change in SSH server, change in authorization technique (i.e. switch from NIS to Kerberos), upgrade to subversion (does this matter?), local changes on the client due to version/other repo action, a foreign hack, albeit fairly non-malicious.

Filed in code, Freeform | Comments (0)

matrix multiplication in mySQL

Friday, May 1, 2009

Background

For a while now, there has been a need in a project of mine to perform a large-scale matrix multiplication. Here, large-scale involves the multiplication of a 2000×2000 square matrix with another matrix containing labels.  A simpler example is projecting a matrix containing a set of class features into a 2-d Cartesian space for easy visualization of a problem.  Although it’s not what is currently in use, this projection was considered a solution for CuZero, one of the research projects that I worked on.

  • S_{n \times c} , where n is the number of samples in the test and c is the number of classes; each row in the matrix is the score of a single sample for all classes
  • P_{c \times d} , where c is still the class count and d is the dimensionality of the new visualization space; for simplicity, this example sets d=2 for a 2-d visualization (below)

While this code could (and is currently) written in PHP, there is a lot of wasted memory and retrieval time getting all column/rows fromthe database into the script.  Additionally, there are wasted cycles from matrix sparsity that results from a similarity matrix, as defined by this top-k optimization post.  So, with this as the backdrop, let’s dig into the data and procedure.

Motivating Example

Okay, to better illustrate the problem, synthesized data is provided.  Matlab was used to do it, but assume that you have data for any problem.  For example, say that you took a poll about how much people liked three sports: baseball, basketball, football.  Now, you want to visualize how the population that you polled can vary their preferences between the three sports in an intuitive 2-d graphic.  The code to make such an example is below and also attached as a set of CSV files; don’t worry about the 0.5 and 0.25 scalars on these equations, they’re just there to make the random data naturally cluster into three groups.

% initialize our projected coordinates
P = [-1 -1; 0 1; 1 0];
% plot the positions (c1=red, c2=green, c3=blue)
colorSets = [1,0,0;0,1,0;0,0,1];
subplot(1,2,1), scatter(P(:,1),P(:,2),70,colorSets, 'x');
axis([-1 1 -1 1]); grid on; title('Initial Positions');
% initialize our set of samples
S = rand(99,3)*0.25;
for idxClass=1:3
   idxRange = [(idxClass-1)*33+1:idxClass*33];               % helper for range
   S(idxRange,idxClass) = S(idxRange, idxClass)*0.5 + 0.5;   % update this column
end

% perform the mapping with a simple matrix multiplication
mapP = S*P;
% plot the mapped positions
subplot(1,2,2), hold on;
for idxClass=1:3
   idxRange = [(idxClass-1)*33+1:idxClass*33];               % helper for range
   scatter(mapP(idxRange,1), mapP(idxRange,2), 40, colorSets(idxClass,:), 'o', 'filled');
end
hold off; axis([-1 1 -1 1]); grid on; title('Projected Features');

Here’s the resulting plot with the original positions plotted as different shapes and in different colors and the mapped samples exclusively plotted as red x’s.  This operation is trivial to do in this environment or with a low dimensionality, but doing it in SQL isn’t so bad because of it’s relational indexing nature.

Projected example values

mySQL implementation

Now let’s move the focus to mySQL.  Assume there are two tables: features, containing the class features of the individual samples and projections containing the visualization coordinates of the classes.  Although the example data above is simple, we must design a schema that is generic for an unlimited number of classes.  For this reason, we design the features table with three columns: id_sample, id_class, and score; the projections table also has three columns: id_class, dimension, position.  Once your data is flattened to this type of structure, it is fairly trivial to execute a matrix multiple, thanks to mySQL’s relational column indexing.

SELECT f1.id_sample, p1.dimension, SUM(f1.score*p1.position) as position
FROM projections AS p1
LEFT JOIN features as f1 ON f1.id_class=p1.id_class
GROUP BY p1.dimension, f1.id_sample

relational multiplication

That’s it.  Surely all your time above wasn’t a waste, right? Well, not exactly. There are several perks that were getting for free just by using the SQL code.  First, we can skip sparse rows/columns because if any sample in features that doesn’t have a row for a given class, the JOIN will skip it.  We’re using a left join, so the other case may not necessarily be true.  Specifically, all classes that exist in the features table really should have a valid row entry in the position table as well.  Don’t believe me? Well, you can download the SQL or raw data files in CSV format to try it on your own.

A Practical Example

If you’ve read this far, you deserve to be rewarded.  As mentioned, the reason for this whole exercise is to do fast matrix multiplications online.  Combined with a very smart method for graph transduction, as was done by a peer, in this presentation and through this work on cell identification and other extensions.  I employed this technique on a subset of 2,000 flickr images crawled sometime in early 2009 using geo-queries and the tags “Columbia University”.   First, please try out the demo below, written with a bit of jQuery, mySQL, and PHP and using fused grid-color moments, gabor texture, and edge direction histogram similarity.  You can get some interesting results by labeling just a few images of buildings with columns or outdoor scenes; it should be noted that the contribution of this process is the low label count required.

  • Change the number of images currently viewed (ordered by decreasing relevance)
  • Modify the number of k-nearest neighbors (discussed below) and see the effect on result quality.
  • Click ‘random‘ to randomly jump around the set and to know that the system isn’t cheating
  • Label an image by clicking (once for positive, twice for negative, three times for unlabeled); Shift-click to view the original image from flickr
  • Click ‘rerank‘ when you’re ready to update the model
  • Click ‘random‘ to get more images or ‘clear‘ to restart the process

Random

Rerank

Clear
Sorry, if you are seeing this text the demo isn’t working in your browser.  Please make sure that you have javascript enabled and try opening the page directly instead of through the feed or search page.

Extensions For Approach

It’s out of the scope of this page to discuss the technical aspects of this algorithm, so please don’t ask for the complete code; it’s all in the paper and presentation and more importantly, you can probably find it on the project page at the bottom.  There are a few problems that may or may not be obvious from this demo and the presentation.

  • Since we’re now dealing with a weight matrix derived from k-NN (again, see the prior discussion), it’s important to just use the top-K members.  Choose a small k, generally 2-5% of the total database size is good enough (i.e. 40-100 nearest neighbors), to populate your weight matrix.
  • Don’t forget to make the matrix symmetric.  Fortunately, this can be done easily in SQL by first creating a temporary table and updating as necessary.

Both of these simple solutions are illustrated below.  In this example we have one similarity table with 4 columns (id1, id2, neighbor, score), one table for the user-provided labels labels with two columns (id1, label), and a temporary table matmult with 3 columns (id1, id2, symmetric) created at runtime.  One important thing to note is that we create a unique index on the combination of both columns.

# compose temporary matrix for N x C
CREATE TEMPORARY TABLE matmult (
    rowI MEDIUMINT NOT NULL,
    colI MEDIUMINT NOT NULL,
    symmetric FLOAT NOT NULL,
    UNIQUE INDEX (rowI, colI) );

# populate the temporary matrix (second stroke make symmetric)
INSERT INTO matmult
    SELECT s1.id1 as rowI, s1.id2 as colI, s1.score as symmetric
    FROM similarity as s1
    WHERE s1.neighbor < $MAX_DEPTH$;
INSERT INTO matmult
    SELECT s1.id2 as rowI, s1.id1 as colI, s1.score as symmetric
    FROM similarity as s1
    WHERE s1.neighbor < $MAX_DEPTH$
    ON DUPLICATE KEY UPDATE symmetric = (s1.score+symmetric)/2

The rest of the operation can be performed normally because the matrix matmult is now symmetric and has valid column/row entries for every available image id.  Another optmization, purely for speed, would be to limit the samples that had valid id1 or id2 values that are within your label set.

Filed in code, research | Comments (0)

Weather Aware Sampling and Comparison of Climate Data: An Alternative Way to Better Climate Prediction

Wednesday, April 29, 2009

Weather Aware Sampling and Comparison of Climate Data: An Alternative Way to Better Climate Prediction

  • speakers: Mike Bauer (NASA GISS, NASA Postdoctoral Fellow), george tselioudis, bill rows, climate research
    • host: Claire Monteleoni, CCLS, CCLS Conference Room, Suite 850, 475 Riverside Dr.
  • abstract: Climate researchers such as myself often face a flurry of questions from friends, family and others following unusual weather events. These questions range from gleeful challenges such as “Where’s global warming now?” to breathless worries of “How bad will it get?” Following a deep sigh the common retort to this is that climatologists study climate not weather and as the old saw goes “weather is not climate.” Of course this is only half-true as climatologists do study weather, only statistically, and weather is indeed a main ingredient of climate, and yes, even those pesky unusual weather events contribute to it.
    • It shouldn’t be surprising then to learn that climate models simulate weather. What may be surprising though is that the correctness of this simulated weather is rarely assessed directly. Instead, traditional methods of model validation rely on long-term averages. Which is consistent with the “weather is not climate” sentiment.
    • An alternative approach to model validation will be resented, one that makes use of our knowledge of weather processes, such as their patterns of occurrence, structure and behavior to test climate models in a new and informative way. In this way we can broaden traditional methods of climate model validation without replacing them, which is to say that we aim to merge the context afforded by the case-by-case perspective of the meteorologist with the statistical vantage of the climatologist. To do this we have a method for identifying, following and delimiting a target weather phenomenon (in this case mid-latitude cyclones). We will show how this tool can be used to identify specific model deficiencies as well as open up new research possibilities.

background

  • climate model – layers, time, variables, model experiments, models – all can be varied and compared independently
  • data – remote sensing, weather balloons, satellites
    • problems: incomplete, sparse, different time and space coverage and interval
    • problem: not forecass, just recordings
  • traditional method – averaging over time series (long-term average to observations)
    • problem: lossy compression, too often/too much, not sure why there is a difference or loss
    • solutions: trying to switch from eulerian to Lagrangian POV (i.e. following a cloud instead its space)

tool demonstration

  • atomic – sample takers that can find a specific time or place, or large domain
  • event-based extraction – using simple thresholds can find certain events
  • phenomenon based extraction – connected events, Lagrangian, specific answers (but must now answer why given too much data)
    • simple partitioning of data to quickly parse unique events

extra-tropical cycles

  • why extra-tropical cyclones as example?
    • easy to find, well understood; characteristic scale but interesting variability; imprint seen in observations and models
    • climate change, feedback and uncertainty (no analog in the past for what will happen in the future)
      • sensitivities not captured in models but need to predict these minor variables for future use
    • one example problem in large amount of data to find characteristics through this filter; example or larger generic class of problem

traditional problems

  • traditional method uses mass of features
  • new method uses mass distribution using sea-level projection
    • advantage: find low-pressure, radial events with Lagrangian ; same size/pressure calculation
  • challenge: find and track — currently just use a filtering criterion
    • ML can do finding faster based on identifying individuals
    • minima in local pressure field; local and regional (characteristic sizes at both scale), laplacian bend of pressure field (second derivative)
    • tracking – similarity and proximity – i.e. simply computed with a general cost function
    • mostly single choice – usually just to connect or not connect a single instance, not multiple points; still a problem because some instances can merge and split (i.e. waves in a fluid, with some supervised criterion)

ML possibilities

  • cyclone attribution – potential for ML
    • partitioning – currently use largest set that encloses center (with iso-pressure line); segregate into cyclone vs. non-cyclone
    • seeded region growing – detection based on edge gradient (doesn’t always work where there is an ‘eye’ in isomap)
      • nesting of centers can occur to distort attribution of grids to a specific depression
      • discrimination can still be achieved with time-based tracking? yes
      • also used Fourier harmonics to do merging and bifurcation (from instabilities)
  • GIS / context/ data sort – potential for ML? (probably database/retrieval)
    • find nearest cyclone given a specific sample or location (i.e. frequency or confidence)
    • could be used for feature selection or for parameter estimation — look at ‘consequence’ of this tracking from real-world elements and differing/detecting model faults
    • given classification, can find feature values from these sets — correlate label to attribute values
  • cyclone composites – quality assessments (i.e. existing cluster analysis)
    • teach/learn a behavior given these weak/moderate/strong labels or a specific place
    • models can be used to both classify and generate cyclone examples
    • develop a taxonomy/lexicon for cyclone classification
    • goal: define a few specific parameters and find main regimes of variability….
      • define in a query-based structure

summary

  • python – open source, mostly OO, sometimes parallel (SMP – multi-core)
  • current applications – climate model validation, satellite data reduction, weather sensitive
    • ecology, oceanography, air quality, wind energy, insurance

questions

  • run models then compare against observations?
    • climate models vs. real analysis (fill in information with a given model) – this real data can be used to find other events
    • usually climatologists would look at aggregate numbers
  • is this task model calibration?
    • yes, but also learning in general based on statistics to identify specific events and average them
  • is radius of cyclone large enough
    • break up world into general grids (geo and location); generally things aren’t “resolved” until you have ~10 grids
    • granularity of grids is typically 100 km^3
  • cyclone vs. hurricane
    • some similarities, hurricanes mostly stay in tropics, cyclones only exist outside typically at 30 deg
    • hurricanes depend on sea temperature (narrow feature band), cyclones come around unstable system in gradient to the poles but it is generally a dynamical instability
    • typically when dissipating will die in the poles; always there but has a seasonal effect
  • combinatorial blow-up for filtering?
    • not exactly, choose local minima after filtering to avoid trying all possible regions
  • is this efficient? or just dynamic programming?
    • nump (python/c sorting), pretty fast to run climate all data in under an hour
  • in terms of extrapolating is it also important to find things where you don’t know the exact properties
    • yes, but prior work finding these patterns is not unique; in the past was very non-robust
    • generally, these solutions will probably still be interesting for traditional meteorology discoveries; also may find patterns because more data is available now vs. then
  • below ‘scale’ phenomena – effects of these may be significant, but miss on data filtering by model scale
    • this problem is a little different, but ML is will suited to this problem; take highly-resolved models (more detail in physics and events) and to derive new models
  • sat data resolves to smaller scales, but could analyze empirical data instead of models?
    • possibly, but sat data is only view from top (or a slice), but may not have wind/temperature information
    • these data sources are typically incomplete, but physical model isn’t really based on truth
    • do better with not constructing based on sat analysis (i.e. need other components that are contributing because observations are sparse)
  • what is the model size of data?
    • model output ~20TB, with other data, can be up to PB; about 1TB a week from “a-train”

terminology

  • parameterize – cube of 100km to discretize atmosphere and data sections
      1. single value then represents that entire region – estimation/generation could be used
  • evaluation not ‘validation’ – models aren’t perfect, better terminology
  • examples: “a-train” follows and can give different instrument readings
    • goal: put all of these together to look at a larger region with these data segments – need to be collected/normalized

summary ideas / suggestions during ‘brainstorming’ — autumn time frame

  • better ways to parameterize within the “grid boxes” – estimate processes that happen below that scale
    • vision could help to do 3d filling of these grid boxes
  • data-mining within large datasets of both model output and real data to improve predictions and model output
  • dealing with predictions of future with non-random class of models when compared to real-world; tracking?
    • for models very sensitive to reaction rates? need better estimates of reaction rates?
      • problem is that parameters within model are not real (i.e. gravity, ozone breakdown, etc)
    • distributed / parallel data mining? considered same problem

subtopics

  • interactive, query-formulation systems could help?
    • may not necessarily apply to this problem
  • query system that can validate both systems (observations and model output)
    • use observation based results and try to sort through these missing parameters
    • alternative is to solve disjoint data between observed/model

interesting links and commentary

Filed in research, Seminars | Comments (0)

Starting and Managing a Software Company

Wednesday, April 15, 2009

paradigm shift in software development

  • speaker: Eric Lazarus
  • abstract
    • Paradigm Shift Software Development (PSSD) is designed to integrate well into the popular software development methodologies such as RUP and Agile and augment them in areas that they are often silent.
    • If you plan to lead a small or large technology intuitive, from building a cool interactive website to managing R&D for a large technology company, this talk will provide practical insight about how to do the work better and produce more powerful technology solutions.
  • bio : AI research / tools -> building ‘impossible’ => ‘can’t fail
    • background and passion for philosophy
    • architect for jp morgan, ibm, etc
    • developed reports of security of voting
    • investigator on NSF contract with NIST
  • background
    • Eric Lazarus has been building and deploying complex and innovative computer systems for some of American’s leading companies including IBM, Prudential, JP Morgan Chase, mega-manufactures including GM and Unilever and mega-retailers including SuperValu for more than 25 years. He has managed R&D activities for such organizations as Stanford University and the National Science Foundation. In this talk he will explain the key insights he has had that has lead him to codifying an unusual and powerful approach to developing innovative technologies in a reliable and dependable way. In this talk he will introduce such ideas as Radical Respectfulness, an approach that encourages managers to give something that costs nothing and get something in return that is priceless: An increased commitment to project success multiplied by the willingness to engage far greater creative energy. He will also discuss the technique called Power Questions which allows you, as a technology manager, to identify issues and opportunities early and use the resources of your entire team and personal network to address them.
  • paradigm shift in software development
    • early insight, just starting out, high tech start up, project in trouble, consultant just wasted time

“meta-empowerment”

  • meta-empowerment was clear lesson – if have managers need to empower them to let those managed understand work
  • reasons for project failure: misunderstanding of requirements, risks, lack of commitment to clear vision of success, missing skills or key resources, most methods do not focus enough on reason for failure
    • one common lack of info: why is budget important, why first version needed for os Z?
    • manager may put goal in risk but can’t find this requirement unless it’s explicitly stated
    • often business side will try to push all requirements at once… but need to only specify a few points that they can optimize
  • first principles of PSSD
    • radical respectfulness – person who devices “clever” technique that saves project may not be from expected source
    • paradigm shift is possible at each level – think of new alternatives at each step in the process; typically might just be innovation during design, but should be always
    • participative – doer decides
    • it is role of leader to provide vision to project – don’t really focus on ‘vision’
      • not all visions equal – “fitness goal” vs. “athletic goal” – athletic goal could be much more powerful
      • motivating, energizing, powerful, charismatic
      • calls all to ‘step-up-to-the-place’
  • what is vision
    • job is … consider visions, pick some, articulate vision

radical respectfulness

  • collaborate with ‘product management’ not be slaves
  • programmers as ‘replaceable parts’ – circle nouns
    • hire very carefully
    • invest in mentoring, building specific skills needed in near term
  • be/express confidence that your team can ID risks
    • work with your team to mitigate risks
  • challenge: cant use all ideas, so what then? DBT – dialect-able behavioral therapy
    • range of minor empathy to all society understands — give most validation possible without mistruth

decision matrix (DM)

  • used in extended groups… – performance, scale, cost, etc. with a weighting factor
    • deeper understanding of project can come from questions and considerations from team
    • options that can be critical to a breakthrough can come from anywhere
    • taking part in developing DM goes far to help people engage in a process
  • how is this helpful?
    • have objective evaluation because agreed upon, decisions aren’t made in haste, i.e. without emotion
    • explore space of options: goals/values, options, understand intersection — anything else is a guess
    • reducing friction – makes manager a peer in the group (requirements are in charge, not manager)
    • covers all assumptions of everyone – may allow non-highest score operation to win
  • when should they be used…
    • answer not obvious, goals are not obvious
    • get participation when – you will be able to use information, overhead is acceptable
    • DO NOT use it if decision should not be left to others
  • method is applicable
    • design decisions, hiring decisions
    • requirement decisions (what features should be in release)
    • architecture
    • risk mitigation
      • no other known project definitions seem to define the risk, probabilities, and results of failures
    • being open to revising DM as assume set of values is essential
  • job as project executive is ….
    • now clearly defined by goals in DM
    • allows you to know the limits of meta-knowledge – “knowledge of knowledge” (wisdom)
      • example: expert in inventory system didn’t understand how it would change with perishables (food)

second principles of PSSD

  • possible to have shift at every stage, preparing to have one is enormously beneficial
  • how to do it
    • ask key questions – meta questions – inspire people to question
    • affirm input, reward it, give credit liberally, renaming it
    • expect it – insight is RIGHT HERE
    • provide CONTEXT
    • hire for it, empower for it
    • openness to tools, techniques, approaches, question assumptions
  • questions (power question examples)
    • how will this project be deemed successful?
    • how can we make you “very happy” that meets all requirements but not happy?
  • challenges that will be faced
    • inertia – people used to projects in a specific way
      • make a ‘role model’ from the audience and show this as an example, more powerful if just you do it
      • reward all acts of initiative (reward pigeon getting close to ball, then you can reward after playing it)
    • lack of meta-knowledge (don’t know that lack of knowledge is here)
      • ask impertinent questions — ask ‘stupid’ questions about why people do certain things
      • highlight insights – point out your own wrong assumption (also good to debug arrogance on your part)
    • belief that “i can’t make a difference in org of this size”
      • disprove and highlight
    • creating resentment / making enemies – might be perceived as criticizing
      • focus on positive both of old way and new; cheerful and give credit

questions

  • how to remove expectation from business side because another can be at risk?
    • show ‘risk’ matrix to business side that was created with developers to show more goals are reached
  • have a list of tools or skills that are standard before hand?
    • start out with use cases (description of features to build) then think about tech to use
  • how do you know that you have all of the needed columns in DM?
    • allow team to change it, if you can work it out that others have contributed will make them more confident
    • example: working on insurance company as architecture consultant, at first seemed okay, but asked insurance people how will you know if it succeeds? this process took a long time to find the simple answer… if an incoming call could be completely diagnosed in one call it was good
  • asking if schedule can be done in 2 months, would it hope to show it done in 1 month
    • similar example is just asking if ‘life depended on it’ what would it take to exceed expectations

technology project discussion

software problem – voice to voice translation

  • idea: translate spoken english into another language – Schmidt says it will be next big thing
    • voice recognition has high error rate
    • server-based translation may not be available, what about on client itself
    • need natural language translation
      • need to be able to learn online to do translation
      • ‘out of sight out of mind’ <= russian-english double => ‘invisible insanity’; ‘mind was wiling but flesh was weak’ <=> ‘vodka was good but meat had spoiled’
  • prototype partially developed already by CU graduate developers
  • problems
    • identify the market – institutional versus consumer
      • largest market is buisiness travelers and tourists
    • how to communicate with market
      • ‘mail order bride’ where you incentivize use with partial sales
    • hard to work with people from diverse – some suggest facebook approach
  • questions: niche market that is cheap to communicate with but advertisers care a lot and want to ad to them – try to approximate profit with these
    • in startups if you can identify a small niche, that might be good enough
    • example: starting a vitamin company, targeting students (hard to communicate with students), but “nero-enhancer” seemed to help people doing power-lifting “body quicken”
  • what is the meta question we should ask?
    • is gov’t question reasonable? UN has need for “get the gist” quality of translation
    • need to clearly define level of functionality and technical challenges
      • i.e. need to show them in feedback as well

commentary and links

Filed in Seminars | Comments (0)