Speech snippit…

Gabriel Silver Memorial Lecture with Kofi Annan

background

• Topic: global climate change
  • Wednesday, September 23, 2009, 12:00 pm, Rotunda, Low Memorial Library
• acceptance of position as a global fellow at Columbia
  • rooted from one of the first members, Eisenhower, appointed before serving as president
  • spoke towards equality and freedom in each country
  • sense of global community, achieved at Columbia is one that is addressed today
• climate change is not the only threat: conflict, famine, disease, poverty — each will worsen with climate change
  • famine will increase as water resources are used, rise of sea levels, and the acceleration of disease
  • more work is needed, not only through humanitarian reports, but also direct change to avoid some impacts that are at a much faster pace than predicted
  • report issued this summer documented impact on thousands in poorest nations

political change is required

• urge leaders of g20 to catalyze movement; they are responsible for the majority of greenhouse emissions
  • agreement about impact of gasses thus far has not lead to needed political momentum
  • post 2012 agreement must be poignant and binding
  • those that are most threatened live in poor countries in asia, africa, or small island states but have contributed less than 2% of the emissions — current requirements cut by 19% by 2050
  • more aggressive and needed goal is to cut by at least 25% by 2040
• other countries feel that economic opportuniteis must not be threatened – correct, but must still abide by reductions
  • fairness dictates that rules should apply to all (i.e. bicycles should not make up for SUV drivers)
• least developed countries deserve incentives for any reduction that can provide through green land use, reforestation, etc
  • need additional funds as well as transfer of resources and low-carbon technologies

financial requirements

• issues that still need to be resolved include cost of adaptation and going forward — sums so far are no where near enough
  • cost of climate change in Africa will raise from 30 billion in 2013, rising to 50-100 Billion through 2020
• funds may come from pricing carbon not only to provide funds but provide a real reason to reduce consumption and production
• need leaders to put aside narrow national interests and focus on long-term well-being of future generations
  • US will need to be the leader (China has overtaken the country as a whole) but per-capita is 4x China and 2x Europe
  • Agrees that Obama needs to take change now, but can not if stopped at home by populous

leadership is required in many places

• leadership is not confined to politics alone, must also come from business aspects
  • both in minimization and investment in technologies abroad; many advances here will help to combat changes in the future
• the academic world can also lead by brining different disciplines together
  • not only bridging different disciplines, but also through social opportunities to utilize climate change opportunities
  • join the ‘tik tok global campaign for climate justice’ that raises awareness and fair debate in Copenhagen

problems to be overcome

• striking comments by Eisenhower was his optimism
• believe if community can come together, it can provide a new page for international cooperation and understanding
  • currently students are first generation to consider themselves ‘citizen of the world’

questions from the audience

• regarding economy, africa must decrease birth rate to achieve economic development in conflict to Pope’s claim of decreasing birth rate
  • climate change is still making this more difficult (drought, food production , floods, spread of diseases)
  • certainly some countries (like Zimbabwe), the lifespan has dropped by more than 10 years, loosing farmers, doctor, and teacher indicating a loss of the present as well as the future
  • measure are being taken both by the government and foreign firms; example is ‘green revolution for africa’
• what is the most effective way for students to raise awareness
  • network and work with friends in groups that work on climate change; set examples yourself
  • generally work to push issue higher in the political agenda
  • showing that progress has been made, science is no longer questioned
• view of US as a leader in climate change
  • world reaction to Obama’s election was very favorable and positive; often considered it a part of the world’s election
  • this issue goes beyond Obama though because it indicates that US and industrialized countries must lead…
    • if largest polluters continue to do so, developing countries will feel it is also there right as they grow
• thoughts on efforts in Africa for leaders to be more accountable to their people in terms of dollars given
  • see development of society that is trying to keep government responsible; impressed with civil societies that have developed, but more is needed
  • people are demanding transparency, fighting in committee, and want honest security forces — but this progress is also from students coming to the US and others that have gone back to affect change
• Nigerian genocide impact — why did you not take that moment to stop the violence?
  • must understand that UN has several forms: the UN in general assembly (pass mandates, etc) and secretariat (implements mandates)
  • in the case of Rwanda, at this time, international community was withdrawing form Somalia… once US withdrew, other industrialized countries followed suit (removing the best trained and best militarized)
  • set the precedent that troops were not available; Clinton apologized, other governments apologized….
    • but when violence was discovered, a few soldiers were killed and other armies pulled out, largely following US examples
  • in this situation, the UN failed; not for just the secretary but the entire community — problem is that response is only made once the problem has erupted, specifically, there is no standing army
    • example is currently impact in Darfour, needed only 18 helicopters for 2 years, but none were given internationally
• anything that university can do to help students get involved?
  • often asked ‘what should young people do to become good global citizens’ — answer is to organize and take it on, even small problems
  • at university level there are other students and professors to help, but initiative must often be taken by the individual
  • problem perhaps more true in developing countries is that after receiving an education, some people have no link to the needs of their society — perhaps vocational training schools would be a better answer
    • problem also with parents who prefer a degree over practical experience for a country
    • could also encourage this opportunity in a university by asking for a course

final addition — a video for climate change

• 1. cover of ‘beds are burning’ by midnight oil — from many different popular communities
• available as a free download starting in October 1st
  • here is a myspace teaser

Connecting to MacOS X VNC from Linux

VNC is pretty fun and useful for remote-connecting machines. One particular usage that has served its purpose is solving a “my computer is broken” call from parents or siblings, inspired by VNC over SSH and SSH port forwarding to avoid those pesky firewalls. Connecting as a client from a Mac, Chicken of the VNC seems to work with little/no fuss for every scenario I had tried. Sure, there are other packages out there, but why use a commercial one that requires installation if something else is open-source and works fine?

For some reason the “screen sharing” option in OSX seems to return some faulty version information or it just doesn’t do a good job of negotiated color depth, etc. The result is that if you try to connect to a machine running the OSX version of VNC named Apple Remote Desktop or Screen Sharing in 10.5.7 (and earlier), the session you open will immediately close. Running VNC Viewer Free Edition 4.1.1 from a Debian distro, this error presented itself. No option on a GUI (i.e. the Terminal Server Client GUI) would work, as it requires the command-line option below (mandating the use of 32bit color).

vncviewer --FullColor=on

Of course, after you make this change and get VNC to work on your linux client, you may realized that it takes a *lot* of bandwidth to get a fluid response, which for me just wasn’t worth it in the end.

Here’s the original thread that eventually shed light on the subject… VNC Thread

I have to start vncviewer with –FullColour=on, else it terminates without showing the display. I think this means it’s using 32 bpp, even though I’ve configured the Mac to use “thousands of colours” (i.e. 16 bpp) in its display settings dialog, so it is wasting half of its network bandwidth.

Subversion rejects check-in with “Authorization Failed”

Another second annoyance was the sudden rejection of SVN check-in actions. If you had a nice, healthy repository that permitted the check-in or update of your tree and suddenly you’re getting rejection notices you might have the same problem:

localhost:cuzero quinone$ svn ci localtest 
emz@'s password: 
svn: Commit failed (details follow):
svn: Authorization failed
svn: Your commit message was left in a temporary file:
svn:    '/private/tmp/temp1/svn-commit.3.tmp'

Although the error itself is of no help, it’s easy to confirm that you have this problem…

• you’re using ssh+svn:// to access your repository
• you can ssh into your server without problems
• you can ‘checkout’ or ‘update’ your local copy without problems
• when you try to commit you get this ambiguous error.

Fortunately, there is a one line fix. Just add the line below to your “/your/repo/dir/conf/svnserve.conf” file.

[general]
anon-access = write

Assuming you already have unix permissions correctly configured to restrict access to that repo directory, you shouldn’t need to have explicit users defined in your svnserve file. Thus, you can allow anonymous write access, which is granted with the above line. Here’s a Subversion fix post that was found after a few hours of digging. It is still puzzling as to why this line was suddenly needed and all but the last possibility has been ruled out: change in SSH server, change in authorization technique (i.e. switch from NIS to Kerberos), upgrade to subversion (does this matter?), local changes on the client due to version/other repo action, a foreign hack, albeit fairly non-malicious.

For a while now, there has been a need in a project of mine to perform a large-scale matrix multiplication. Here, large-scale involves the multiplication of a 2000×2000 square matrix with another matrix containing labels.  A simpler example is projecting a matrix containing a set of class features into a 2-d Cartesian space for easy visualization of a problem.  Although it’s not what is currently in use, this projection was considered a solution for CuZero, one of the research projects that I worked on.

• , where is the number of samples in the test and is the number of classes; each row in the matrix is the score of a single sample for all classes
• , where is still the class count and is the dimensionality of the new visualization space; for simplicity, this example sets for a 2-d visualization (below)

While this code could (and is currently) written in PHP, there is a lot of wasted memory and retrieval time getting all column/rows fromthe database into the script.  Additionally, there are wasted cycles from matrix sparsity that results from a similarity matrix, as defined by this top-k optimization post.  So, with this as the backdrop, let’s dig into the data and procedure.

Motivating Example

Okay, to better illustrate the problem, synthesized data is provided.  Matlab was used to do it, but assume that you have data for any problem.  For example, say that you took a poll about how much people liked three sports: baseball, basketball, football.  Now, you want to visualize how the population that you polled can vary their preferences between the three sports in an intuitive 2-d graphic.  The code to make such an example is below and also attached as a set of CSV files; don’t worry about the 0.5 and 0.25 scalars on these equations, they’re just there to make the random data naturally cluster into three groups.

% initialize our projected coordinates
P = [-1 -1; 0 1; 1 0];
% plot the positions (c1=red, c2=green, c3=blue)
colorSets = [1,0,0;0,1,0;0,0,1];
subplot(1,2,1), scatter(P(:,1),P(:,2),70,colorSets, 'x');
axis([-1 1 -1 1]); grid on; title('Initial Positions');

% initialize our set of samples
S = rand(99,3)*0.25;
for idxClass=1:3
   idxRange = [(idxClass-1)*33+1:idxClass*33];               % helper for range
   S(idxRange,idxClass) = S(idxRange, idxClass)*0.5 + 0.5;   % update this column
end

% perform the mapping with a simple matrix multiplication
mapP = S*P;
% plot the mapped positions
subplot(1,2,2), hold on;
for idxClass=1:3
   idxRange = [(idxClass-1)*33+1:idxClass*33];               % helper for range
   scatter(mapP(idxRange,1), mapP(idxRange,2), 40, colorSets(idxClass,:), 'o', 'filled');
end
hold off; axis([-1 1 -1 1]); grid on; title('Projected Features');

Here’s the resulting plot with the original positions plotted as different shapes and in different colors and the mapped samples exclusively plotted as red x’s.  This operation is trivial to do in this environment or with a low dimensionality, but doing it in SQL isn’t so bad because of it’s relational indexing nature.

mySQL implementation

Now let’s move the focus to mySQL.  Assume there are two tables: features, containing the class features of the individual samples and projections containing the visualization coordinates of the classes.  Although the example data above is simple, we must design a schema that is generic for an unlimited number of classes.  For this reason, we design the features table with three columns: id_sample, id_class, and score; the projections table also has three columns: id_class, dimension, position.  Once your data is flattened to this type of structure, it is fairly trivial to execute a matrix multiple, thanks to mySQL’s relational column indexing.

SELECT f1.id_sample, p1.dimension, SUM(f1.score*p1.position) as position
FROM projections AS p1
LEFT JOIN features as f1 ON f1.id_class=p1.id_class
GROUP BY p1.dimension, f1.id_sample

That’s it.  Surely all your time above wasn’t a waste, right? Well, not exactly. There are several perks that were getting for free just by using the SQL code.  First, we can skip sparse rows/columns because if any sample in features that doesn’t have a row for a given class, the JOIN will skip it.  We’re using a left join, so the other case may not necessarily be true.  Specifically, all classes that exist in the features table really should have a valid row entry in the position table as well.  Don’t believe me? Well, you can download the SQL or raw data files in CSV format to try it on your own.

A Practical Example

If you’ve read this far, you deserve to be rewarded.  As mentioned, the reason for this whole exercise is to do fast matrix multiplications online.  Combined with a very smart method for graph transduction, as was done by a peer, in this presentation and through this work on cell identification and other extensions.  I employed this technique on a subset of 2,000 flickr images crawled sometime in early 2009 using geo-queries and the tags “Columbia University”.   First, please try out the demo below, written with a bit of jQuery, mySQL, and PHP and using fused grid-color moments, gabor texture, and edge direction histogram similarity.  You can get some interesting results by labeling just a few images of buildings with columns or outdoor scenes; it should be noted that the contribution of this process is the low label count required.

• Change the number of images currently viewed (ordered by decreasing relevance)
• Modify the number of k-nearest neighbors (discussed below) and see the effect on result quality.
• Click ‘random‘ to randomly jump around the set and to know that the system isn’t cheating
• Label an image by clicking (once for positive, twice for negative, three times for unlabeled); Shift-click to view the original image from flickr
• Click ‘rerank‘ when you’re ready to update the model
• Click ‘random‘ to get more images or ‘clear‘ to restart the process

Extensions For Approach

It’s out of the scope of this page to discuss the technical aspects of this algorithm, so please don’t ask for the complete code; it’s all in the paper and presentation and more importantly, you can probably find it on the project page at the bottom.  There are a few problems that may or may not be obvious from this demo and the presentation.

• Since we’re now dealing with a weight matrix derived from k-NN (again, see the prior discussion), it’s important to just use the top-K members.  Choose a small k, generally 2-5% of the total database size is good enough (i.e. 40-100 nearest neighbors), to populate your weight matrix.
• Don’t forget to make the matrix symmetric.  Fortunately, this can be done easily in SQL by first creating a temporary table and updating as necessary.

Both of these simple solutions are illustrated below.  In this example we have one similarity table with 4 columns (id1, id2, neighbor, score), one table for the user-provided labels labels with two columns (id1, label), and a temporary table matmult with 3 columns (id1, id2, symmetric) created at runtime.  One important thing to note is that we create a unique index on the combination of both columns.

# compose temporary matrix for N x C
CREATE TEMPORARY TABLE matmult (
    rowI MEDIUMINT NOT NULL,
    colI MEDIUMINT NOT NULL,
    symmetric FLOAT NOT NULL,
    UNIQUE INDEX (rowI, colI) );

# populate the temporary matrix (second stroke make symmetric)
INSERT INTO matmult
    SELECT s1.id1 as rowI, s1.id2 as colI, s1.score as symmetric
    FROM similarity as s1
    WHERE s1.neighbor < $MAX_DEPTH$;
INSERT INTO matmult
    SELECT s1.id2 as rowI, s1.id1 as colI, s1.score as symmetric
    FROM similarity as s1
    WHERE s1.neighbor < $MAX_DEPTH$
    ON DUPLICATE KEY UPDATE symmetric = (s1.score+symmetric)/2

The rest of the operation can be performed normally because the matrix matmult is now symmetric and has valid column/row entries for every available image id.  Another optmization, purely for speed, would be to limit the samples that had valid id1 or id2 values that are within your label set.

• speakers: Mike Bauer (NASA GISS, NASA Postdoctoral Fellow), george tselioudis, bill rows, climate research
  • host: Claire Monteleoni, CCLS, CCLS Conference Room, Suite 850, 475 Riverside Dr.
• abstract: Climate researchers such as myself often face a flurry of questions from friends, family and others following unusual weather events. These questions range from gleeful challenges such as “Where’s global warming now?” to breathless worries of “How bad will it get?” Following a deep sigh the common retort to this is that climatologists study climate not weather and as the old saw goes “weather is not climate.” Of course this is only half-true as climatologists do study weather, only statistically, and weather is indeed a main ingredient of climate, and yes, even those pesky unusual weather events contribute to it.
  • It shouldn’t be surprising then to learn that climate models simulate weather. What may be surprising though is that the correctness of this simulated weather is rarely assessed directly. Instead, traditional methods of model validation rely on long-term averages. Which is consistent with the “weather is not climate” sentiment.
  • An alternative approach to model validation will be resented, one that makes use of our knowledge of weather processes, such as their patterns of occurrence, structure and behavior to test climate models in a new and informative way. In this way we can broaden traditional methods of climate model validation without replacing them, which is to say that we aim to merge the context afforded by the case-by-case perspective of the meteorologist with the statistical vantage of the climatologist. To do this we have a method for identifying, following and delimiting a target weather phenomenon (in this case mid-latitude cyclones). We will show how this tool can be used to identify specific model deficiencies as well as open up new research possibilities.

background

• climate model – layers, time, variables, model experiments, models – all can be varied and compared independently
• data – remote sensing, weather balloons, satellites
  • problems: incomplete, sparse, different time and space coverage and interval
  • problem: not forecass, just recordings
• traditional method – averaging over time series (long-term average to observations)
  • problem: lossy compression, too often/too much, not sure why there is a difference or loss
  • solutions: trying to switch from eulerian to Lagrangian POV (i.e. following a cloud instead its space)

tool demonstration

• atomic – sample takers that can find a specific time or place, or large domain
• event-based extraction – using simple thresholds can find certain events
• phenomenon based extraction – connected events, Lagrangian, specific answers (but must now answer why given too much data)
  • simple partitioning of data to quickly parse unique events

extra-tropical cycles

• why extra-tropical cyclones as example?
  • easy to find, well understood; characteristic scale but interesting variability; imprint seen in observations and models
  • climate change, feedback and uncertainty (no analog in the past for what will happen in the future)
    • sensitivities not captured in models but need to predict these minor variables for future use
  • one example problem in large amount of data to find characteristics through this filter; example or larger generic class of problem

traditional problems

• traditional method uses mass of features
• new method uses mass distribution using sea-level projection
  • advantage: find low-pressure, radial events with Lagrangian ; same size/pressure calculation
• challenge: find and track — currently just use a filtering criterion
  • ML can do finding faster based on identifying individuals
  • minima in local pressure field; local and regional (characteristic sizes at both scale), laplacian bend of pressure field (second derivative)
  • tracking – similarity and proximity – i.e. simply computed with a general cost function
  • mostly single choice – usually just to connect or not connect a single instance, not multiple points; still a problem because some instances can merge and split (i.e. waves in a fluid, with some supervised criterion)

ML possibilities

• cyclone attribution – potential for ML
  • partitioning – currently use largest set that encloses center (with iso-pressure line); segregate into cyclone vs. non-cyclone
  • seeded region growing – detection based on edge gradient (doesn’t always work where there is an ‘eye’ in isomap)
    • nesting of centers can occur to distort attribution of grids to a specific depression
    • discrimination can still be achieved with time-based tracking? yes
    • also used Fourier harmonics to do merging and bifurcation (from instabilities)
• GIS / context/ data sort – potential for ML? (probably database/retrieval)
  • find nearest cyclone given a specific sample or location (i.e. frequency or confidence)
  • could be used for feature selection or for parameter estimation — look at ‘consequence’ of this tracking from real-world elements and differing/detecting model faults
  • given classification, can find feature values from these sets — correlate label to attribute values
• cyclone composites – quality assessments (i.e. existing cluster analysis)
  • teach/learn a behavior given these weak/moderate/strong labels or a specific place
  • models can be used to both classify and generate cyclone examples
  • develop a taxonomy/lexicon for cyclone classification
  • goal: define a few specific parameters and find main regimes of variability….
    • define in a query-based structure

summary

• python – open source, mostly OO, sometimes parallel (SMP – multi-core)
• current applications – climate model validation, satellite data reduction, weather sensitive
  • ecology, oceanography, air quality, wind energy, insurance

questions

• run models then compare against observations?
  • climate models vs. real analysis (fill in information with a given model) – this real data can be used to find other events
  • usually climatologists would look at aggregate numbers
• is this task model calibration?
  • yes, but also learning in general based on statistics to identify specific events and average them
• is radius of cyclone large enough
  • break up world into general grids (geo and location); generally things aren’t “resolved” until you have ~10 grids
  • granularity of grids is typically 100 km^3
• cyclone vs. hurricane
  • some similarities, hurricanes mostly stay in tropics, cyclones only exist outside typically at 30 deg
  • hurricanes depend on sea temperature (narrow feature band), cyclones come around unstable system in gradient to the poles but it is generally a dynamical instability
  • typically when dissipating will die in the poles; always there but has a seasonal effect
• combinatorial blow-up for filtering?
  • not exactly, choose local minima after filtering to avoid trying all possible regions
• is this efficient? or just dynamic programming?
  • nump (python/c sorting), pretty fast to run climate all data in under an hour
• in terms of extrapolating is it also important to find things where you don’t know the exact properties
  • yes, but prior work finding these patterns is not unique; in the past was very non-robust
  • generally, these solutions will probably still be interesting for traditional meteorology discoveries; also may find patterns because more data is available now vs. then
• below ‘scale’ phenomena – effects of these may be significant, but miss on data filtering by model scale
  • this problem is a little different, but ML is will suited to this problem; take highly-resolved models (more detail in physics and events) and to derive new models
• sat data resolves to smaller scales, but could analyze empirical data instead of models?
  • possibly, but sat data is only view from top (or a slice), but may not have wind/temperature information
  • these data sources are typically incomplete, but physical model isn’t really based on truth
  • do better with not constructing based on sat analysis (i.e. need other components that are contributing because observations are sparse)
• what is the model size of data?
  • model output ~20TB, with other data, can be up to PB; about 1TB a week from “a-train”

terminology

• parameterize – cube of 100km to discretize atmosphere and data sections
  • 1. single value then represents that entire region – estimation/generation could be used
• evaluation not ‘validation’ – models aren’t perfect, better terminology
• examples: “a-train” follows and can give different instrument readings
  • goal: put all of these together to look at a larger region with these data segments – need to be collected/normalized

summary ideas / suggestions during ‘brainstorming’ — autumn time frame

• better ways to parameterize within the “grid boxes” – estimate processes that happen below that scale
  • vision could help to do 3d filling of these grid boxes
• data-mining within large datasets of both model output and real data to improve predictions and model output
• dealing with predictions of future with non-random class of models when compared to real-world; tracking?
  • for models very sensitive to reaction rates? need better estimates of reaction rates?
    • problem is that parameters within model are not real (i.e. gravity, ozone breakdown, etc)
  • distributed / parallel data mining? considered same problem

subtopics

• interactive, query-formulation systems could help?
  • may not necessarily apply to this problem
• query system that can validate both systems (observations and model output)
  • use observation based results and try to sort through these missing parameters
  • alternative is to solve disjoint data between observed/model

interesting links and commentary

• available data for download?
  • WeatherUnderground
  • NOAA brochure information
  • National Climactic Data Center download of official weather data, but here is the free data
    • Gridded data available from NCDC
  • NASA ATrain historical data download
• types of existing models
  • aTrain Visulzation on google earth, looks like a conference poster
• possible research in the area
  • google scholar search for climatology articles

• speaker: Eric Lazarus
• abstract
  • Paradigm Shift Software Development (PSSD) is designed to integrate well into the popular software development methodologies such as RUP and Agile and augment them in areas that they are often silent.
  • If you plan to lead a small or large technology intuitive, from building a cool interactive website to managing R&D for a large technology company, this talk will provide practical insight about how to do the work better and produce more powerful technology solutions.
• bio : AI research / tools -> building ‘impossible’ => ‘can’t fail
  • background and passion for philosophy
  • architect for jp morgan, ibm, etc
  • developed reports of security of voting
  • investigator on NSF contract with NIST
• background
  • Eric Lazarus has been building and deploying complex and innovative computer systems for some of American’s leading companies including IBM, Prudential, JP Morgan Chase, mega-manufactures including GM and Unilever and mega-retailers including SuperValu for more than 25 years. He has managed R&D activities for such organizations as Stanford University and the National Science Foundation. In this talk he will explain the key insights he has had that has lead him to codifying an unusual and powerful approach to developing innovative technologies in a reliable and dependable way. In this talk he will introduce such ideas as Radical Respectfulness, an approach that encourages managers to give something that costs nothing and get something in return that is priceless: An increased commitment to project success multiplied by the willingness to engage far greater creative energy. He will also discuss the technique called Power Questions which allows you, as a technology manager, to identify issues and opportunities early and use the resources of your entire team and personal network to address them.
• paradigm shift in software development
  • early insight, just starting out, high tech start up, project in trouble, consultant just wasted time

“meta-empowerment”

• meta-empowerment was clear lesson – if have managers need to empower them to let those managed understand work
• reasons for project failure: misunderstanding of requirements, risks, lack of commitment to clear vision of success, missing skills or key resources, most methods do not focus enough on reason for failure
  • one common lack of info: why is budget important, why first version needed for os Z?
  • manager may put goal in risk but can’t find this requirement unless it’s explicitly stated
  • often business side will try to push all requirements at once… but need to only specify a few points that they can optimize
• first principles of PSSD
  • radical respectfulness – person who devices “clever” technique that saves project may not be from expected source
  • paradigm shift is possible at each level – think of new alternatives at each step in the process; typically might just be innovation during design, but should be always
  • participative – doer decides
  • it is role of leader to provide vision to project – don’t really focus on ‘vision’
    • not all visions equal – “fitness goal” vs. “athletic goal” – athletic goal could be much more powerful
    • motivating, energizing, powerful, charismatic
    • calls all to ‘step-up-to-the-place’
• what is vision
  • job is … consider visions, pick some, articulate vision

radical respectfulness

• collaborate with ‘product management’ not be slaves
• programmers as ‘replaceable parts’ – circle nouns
  • hire very carefully
  • invest in mentoring, building specific skills needed in near term
• be/express confidence that your team can ID risks
  • work with your team to mitigate risks
• challenge: cant use all ideas, so what then? DBT – dialect-able behavioral therapy
  • range of minor empathy to all society understands — give most validation possible without mistruth

decision matrix (DM)

• used in extended groups… – performance, scale, cost, etc. with a weighting factor
  • deeper understanding of project can come from questions and considerations from team
  • options that can be critical to a breakthrough can come from anywhere
  • taking part in developing DM goes far to help people engage in a process
• how is this helpful?
  • have objective evaluation because agreed upon, decisions aren’t made in haste, i.e. without emotion
  • explore space of options: goals/values, options, understand intersection — anything else is a guess
  • reducing friction – makes manager a peer in the group (requirements are in charge, not manager)
  • covers all assumptions of everyone – may allow non-highest score operation to win
• when should they be used…
  • answer not obvious, goals are not obvious
  • get participation when – you will be able to use information, overhead is acceptable
  • DO NOT use it if decision should not be left to others
• method is applicable
  • design decisions, hiring decisions
  • requirement decisions (what features should be in release)
  • architecture
  • risk mitigation
    • no other known project definitions seem to define the risk, probabilities, and results of failures
  • being open to revising DM as assume set of values is essential
• job as project executive is ….
  • now clearly defined by goals in DM
  • allows you to know the limits of meta-knowledge – “knowledge of knowledge” (wisdom)
    • example: expert in inventory system didn’t understand how it would change with perishables (food)

second principles of PSSD

• possible to have shift at every stage, preparing to have one is enormously beneficial
• how to do it
  • ask key questions – meta questions – inspire people to question
  • affirm input, reward it, give credit liberally, renaming it
  • expect it – insight is RIGHT HERE
  • provide CONTEXT
  • hire for it, empower for it
  • openness to tools, techniques, approaches, question assumptions
• questions (power question examples)
  • how will this project be deemed successful?
  • how can we make you “very happy” that meets all requirements but not happy?
• challenges that will be faced
  • inertia – people used to projects in a specific way
    • make a ‘role model’ from the audience and show this as an example, more powerful if just you do it
    • reward all acts of initiative (reward pigeon getting close to ball, then you can reward after playing it)
  • lack of meta-knowledge (don’t know that lack of knowledge is here)
    • ask impertinent questions — ask ‘stupid’ questions about why people do certain things
    • highlight insights – point out your own wrong assumption (also good to debug arrogance on your part)
  • belief that “i can’t make a difference in org of this size”
    • disprove and highlight
  • creating resentment / making enemies – might be perceived as criticizing
    • focus on positive both of old way and new; cheerful and give credit

questions

• how to remove expectation from business side because another can be at risk?
  • show ‘risk’ matrix to business side that was created with developers to show more goals are reached
• have a list of tools or skills that are standard before hand?
  • start out with use cases (description of features to build) then think about tech to use
• how do you know that you have all of the needed columns in DM?
  • allow team to change it, if you can work it out that others have contributed will make them more confident
  • example: working on insurance company as architecture consultant, at first seemed okay, but asked insurance people how will you know if it succeeds? this process took a long time to find the simple answer… if an incoming call could be completely diagnosed in one call it was good
• asking if schedule can be done in 2 months, would it hope to show it done in 1 month
  • similar example is just asking if ‘life depended on it’ what would it take to exceed expectations

technology project discussion

software problem – voice to voice translation

• idea: translate spoken english into another language – Schmidt says it will be next big thing
  • voice recognition has high error rate
  • server-based translation may not be available, what about on client itself
  • need natural language translation
    • need to be able to learn online to do translation
    • ‘out of sight out of mind’ <= russian-english double => ‘invisible insanity’; ‘mind was wiling but flesh was weak’ <=> ‘vodka was good but meat had spoiled’
• prototype partially developed already by CU graduate developers
• problems
  • identify the market – institutional versus consumer
    • largest market is buisiness travelers and tourists
  • how to communicate with market
    • ‘mail order bride’ where you incentivize use with partial sales
  • hard to work with people from diverse – some suggest facebook approach
• questions: niche market that is cheap to communicate with but advertisers care a lot and want to ad to them – try to approximate profit with these
  • in startups if you can identify a small niche, that might be good enough
  • example: starting a vitamin company, targeting students (hard to communicate with students), but “nero-enhancer” seemed to help people doing power-lifting “body quicken”
• what is the meta question we should ask?
  • is gov’t question reasonable? UN has need for “get the gist” quality of translation
  • need to clearly define level of functionality and technical challenges
    • i.e. need to show them in feedback as well

commentary and links

• popular management strategies
  • six sigma and its root cause analysis techniques are similar to those mentioned here
  • Agile and it’s short-cycle version of development called a scrum.
• other software / development models
  • rational rose now owned by IBM

Looking at print media specifically and one of the ways that I interface with news, several audio reports about adaptations of papers for ultra-local blogs and the criticisms of structural mismanagement. While these criticisms are logically well-thought, they mostly refer to problems in the past, not solutions to the future (excuse my temporary exclusion of hyper-local reporting). So, one simple idea is to look to one on a traditional cliche and heed its advice: “if you can’t beat them, join ‘em” — cue web spam here, here, and even here.

So here it is: express the charge of modern reporters through modern media. Feel free to grab the PNG or PDF, just give me a shout or link back if you do.

• This proposal uses a number of online opportunities to involve users (i.e. readers), opportunistically delay search engines from being the end-all aggregators, and increasing exposure of “user-important” content within a publication.
• Of course, it also moves in the direction of facilitating the online printing and release of all publications — the key difference is to get people interested in an investigative topic early and keep them interested by valuing their suggestions and commentary.
• While changes proposed with this form of publication will require some changes (i.e. releasing preliminary materials, gathering more A/V source material and releasing it), these changes will be welcoming in the aggregation and processing of media that is in store for us in the future.
• Also, assuming the burden of information management can be removed from the technical staff at a publication (i.e. the back-office can be pushed to online databases and data management clouds), then costs for the production,  development, and even psychological exposure of new articles can be reduced. Simultaneously, financial support will stay where it’s needed: with the journalists, writers, and editors.
• Public edification and education should really be the target of any publication.  In academic environments, there is now a strong push for open access of publications.  While this market is still going strong (recessions can not effect the need for cutting-edge, peer-reviewed research applications), the online availability of content and needs of those without infinite financial support will erode the sometimes excessive fees of classic journal and conference publications (both to have something printed and to read something you had printed).

Update:

Recently, another article was published and commented on at slashdot that suggests the binding news and entertainment more closely.  This is another example embracing new-media that may push traditional media into the hands of a new audience.  Of course, all of these changes need to be considered in an even-handed manner to avoid the shortfalls of the popular critique of blog-reporters versus traditional reporters.

“As traditional news media struggles to find a new method and business model for dissemination over the internet, some are suggesting that news-related games could be an avenue worth pursuing. Rather than using such games solely as entertainment, journalists could make some of their reports more educative and interactive, allowing readers to choose which threads of a story they would like to follow. Georgia Tech is currently running a research blog to better understand how games and journalism can interact.”

• Speaker: Ted Willke
• Abstract – Presently, aircraft separation and runway usage are controlled manually, primarily through voice communications between Air Traffic Control and pilots. With existing procedures, the air system is suffering from limited runway capacity and airports, such as John F. Kennedy International in New York, and authorities are contemplating construction of an additional airport. Capacity aside, safety remains of paramount concern and yet runway incursion rates have increased steadily over the past two decades. We apply a broadcast protocol, called MRBP, to air-air communications to increase peak capacity by up to 65% for one runway and 49% for intersecting runways. The increases result from aircraft tightly coordinating their movements through shared clearances and other announcements. This shared communication also permits onboard computers to quickly detect aircraft deviations, controller errors, and communication faults. We assess the runway capacity benefits, demonstrate the protocol’s safety mechanisms, and discuss how the protocol can bring the safety benefits of controlled airspace to traffic at smaller regional airports.

background

• worst airport delays in the county
  • jfk: 811/hr (100/r desired), lga: 71, 80, EWR (81 vs 100)
• how do we fix it?
  • either new airport or new technology
  • new runways does little, better surveillance didn’t help, capacity 233 to 280+ is desired
  • propose: wireless communication allowing ATC to be offloaded and decentralized – benefit all 3 NTSB challenges
• systems today
  • networks: ground, approach, departure, tower – departure/approach has independent runways
    • enter queue for different types and handled by FIFO
  • centralized control of points impacts runway – efficiency (timeline decisions and delays), safety (miscommunication, disorientation, non-compliance)
  • responsibility: approach, handoff, spacing, sequencing, traffic routing, runway interference, compliance monitoring, instruction/clearance delivery, runway configuration
    • often don’t deliver message to aircraft until right before because of other jobs

proposed system — collaborative communications

• delegate routine decisions to aircraft to inboard flight systems
  • cohorts maintain a shared view of traffic, requires communication supporting data consistency, with proper inputs distributed will agree on actions in fail-safe
  • several different groups
  • ATC refocuses on system management – queue count, management
• responsibility
  
  • offloaded – spacing execution, runway sequencing, clearance and instruction execution, handoffs, compliance monitoring
  • system management – queue management, sequencing management, traffic routing, runway reconfiguring, exception handling

broadcast protocol (framework)

• mobile reliable broadcast protocol – wireless medium is weakest link and requires new safeguards
  • data consistency – cohorts use same data, two aircraft will use same data in same order
    • message transmit – ACK or retry
    • if majority does not receive, throw away message wait for received – need some safe exit state
    • if majority does receive, consensus on use of data
    • aircraft can discover message was sent but doesn’t have it so can go into safe state for recovery (all others want to use message)
  • deterministic reliability – messages are retried and acknowledged, sources and recipients know when they are at risk of not getting a message

token ring (protocol)

• take a slot in the token ring and own it at a time, that aircraft is responsible for ACK’ing messages at any time
  • earliest site to ACK a message determines its location in ordered list (ACK assigns sequence number)
• any receiver may retry an ACK or message k times
• tokens transmitted after certain time (accounting for network delay) or retry count list the token as received or not [identify equipment failure]
  • future tokens form a vote on whether to include the ACK’d messages in the length
  • vote ends after n tokens are passed
• timeline: message from source -> other sites ACK -> votes shared and agreed upon (vote out of group) -> survivor ACK transmitted
  • create frames of collaboration and it can change over time
  • L1 provides probabilistic reliability exceeding ADS-B
  • L2 provides new data consistency and determinism guarantees
• example 1: aircraft message doesn’t go to ATC, but cohort sends to ATC that one was missing; group accepts or reject based on voting
• example 2: message and ACK loss due to equipment failure; aircraft either recovers with protocol or knows there is a failure because of no messages sent here

applications to capacity

• today: interval starts when A reaches T and ends when D rolls away from T; D is cleared after wake passes and A is assured to exit, release of D from
  • runway occupancy time =
• today: departure arrival – A may arrive too early from G, ATC must ensure A does not overrun D to make sure runway is utilized, D must commit to take off before A commits
  • aircraft/hour
• what can be done:
  • 
  • 70′s – high speed ramps to reduce ROT, 9′s model prediction of W, 00′s – decrease due to improved ATC surveillance tech
  • remains – reduce inter aircraft arrival time, improve time line of clearances
• arrival occupancy model – try to get about 60s for arrivals (observations show that often empty runways)
  • ROT (runway occupancy time) will fluctuate – variations in dynamics, weather, multiple exits
  • solution should provide fine grain control of capacity, reduce ATC workload, increase time for reaction

examples

• simplest method – aircraft following
  • ATC sets following policy, follower motion based on predecessor ADS-B, similar to automated cruise control for road vehicles
  • simulation: tell aircraft to follow, then perturb aircraft at front (for arrival time) speed changes too much
  • simulation: tell aircraft to follow with look-ahead spacing grows more; more stable sequence
• MRBP’s role in approach following
  • establishes group membership, ensures cohorts change leader in concert (synchronous), detect and deal with failures (detect failure, group partitions or falls back to safe)

clearance offload

• today: ATC manually or automatically establishes right-of-way order
  • future: ATC gives policy (management of queue) and cohorts manage actual group operations; reduce workload
• take-off of aircraft can send trigger to cohorts instead of waiting for ATC reaction
• studies have shown that second “crossing” runway does not increase capacity
• lessons – 280+ hurdle will avoid 4th airport
  • JFK – red or blue 100+/hr, LGA – (single — 76.3/hr), EWR blue (100+/hr)

safety improvements

• requires best of bad situation – failure cases are aircraft violates clearance or ATC sends wrong clearance
• micro-flight plan proposed…
  • finer granularity and extend to ground operations
  • reflect performance characteristics of aircraft
  • include machine-generated extrapolation and ETA
  • include annotations of clearance status
• consistency check
  • ATC – clearance != current, detect ATC clearance error, message enabled clearance
  • movement != current clearance or micro-FP – aircraft clearance violation or crew deviation (deviation), message uses clearances, ADS-B, micro-FP
  • revised micro-FP != ATC instructions ….
• summary – collaborative MRBP vs. CPDLC (control-pilot data link communication)
  • makes sure it’s consistent, appropriate, and properly understood
  • performance – make a timely decision, communicate without delay and reliably (already in CPDLC)
  • capacity and safety benefits of new tech are underestimated, stronger communication guarantees new opportunity, ATC distribution imp

questions

• assumption is that runways aren’t used efficiently largely to communication delays, but when allowing aircraft to take-off right after landing, how can this be more quickly communicated?

  • w.r.t. turbulence, can use LIDAR to determine air-patterns and improve detection of weight turbulence and wind-shear; could automate it once sensors are deployed
  • sometimes you can’t see the aircraft, but not counting on crew of waiting operation, counting on aircraft that is doing the action, which should know dynamically and mechanically
  • even save 10-20 seconds for each aircraft will add up over many different operators

• what about deliberate mis-communication by rouge aircraft? (are there fail-safes from other aircraft?)

  • by mandate it’s too severe to interfere with this signal, could add digital signatures and security not really addressed
• what are similar methods deployed at other airports that are comparable? you mentioned international airports as example
  • new technology like “hold-short” lights to prevent violation or collision, new surveillance technology is getting better but still unreliable and gets lost
  • loss is that european airports allow multiple aircraft entering runway
• talked about micro-FP to solve runway problem, would variation of TCAS help to solve this too?
  • info: TCAS (terrain/traffic collision avoidance system) – if two aircraft realize too close then can enter into conflict negotiation between peers…..
  • could, but focused on just that problem (just collision) — could add and use TCAS but part of the problem is that it’s own between two parties, but this lagged-paired reaction could have some critical error
• pointed out that policy might help loss of ATC policy vs. aircraft trying to manage it themselves (i.e. paying attention to ABORT operations)
  • FAA changed definition only recently to define incursion as *possible* problem because of error in decision

interesting links and commentary

• Publications (not vetted)
  • Collaborative automation systems for enhancing airport surface traffic efficiency and safety
  • Representing collaborative work: the airport as common information space
  • Air traffic management: evolution with technology – TACS discussion
• Acronyms
  • What is TCAS?
  • What is CPDLC?
• Related
  • FlightStats – look-up stats on existing flights in the air and real-time status of a specific flight
• News
  • Pilots must now retire at age 65, Dec ’07 FAA
  • Greenwashing the FAA and ATC ABC story
    • statistically, this pollution is about 3% with expectations to grow but may not be too much compared to other commuter/commercial vehicles, also, less dramatic changes within the system like route management can help to reduce fuel consumption
    • 1. danger or reducing the amount of fuel (and therefore weight) for a flight is running short in the case of an emergency which could lead to landing in the wrong place and therefore more delays and dissatisfaction
    • larger ‘comfort’ issues may be addressed first (or in parallel), like making aircraft more silent or both Quiet + eco friendly

Although these notes are almost verbatim of the author’s slides or presentation, where possible, other links in the area and definitions of relevant acronyms will be included. Hopefully, this will allow these posts to be uniquely informative and interesting instead of a mere regurgitation of the original author’s insight. Of course, if you are one of these authors and object to the posting of your material, just send me a quick note and I’ll be happy to take it down.

Enjoy or tell me how you could…

• Abstract: A discussion of how scholars and researchers can take full advantage of opportunities afforded by digital technology in today’s legal environment, and advocate for positive change.This panel is part of the series Research without Borders: The Changing World of Scholarly Communication.
• Location: Intercultural Resource Center (IRC), 2nd Floor, 552 West 114th Street, MC 5755
• Speakers
  • Kenneth Crews, Director of Columbia’s Copyright Advisory Office
  • Michael Carroll, Visiting Professor of Law at American University’s Washington College of Law, a founding member of the Board of Directors of Creative Commons
  • Heather Joseph, Executive Director of the Scholarly Publishing and Academic Resources Coalition(SPARC) and spokesperson for the Alliance for Taxpayer Access

Who really owns your scholarly work… (Kenneth Crews)

• who cares —
  • you do (advancement of good work, future of scholarship),
  • publishers (economic models of publishing, growth and survival)
  • libraries (escalating costs, mission of information access)
• why now – new opportunities for dissemination, control, confusion; new formats, multidirectional communication
  • new players – library as publisher, reader as author, public as investor, google;
  • changing copyright control and importance of contracts – increased copyright protection
• copy right law – legal scene
  • protects nearly everything – original works, fixed in a tangible medium
  • automatic protection – no requirement of formalities, protection for long term (life of author + 70 years) and broad scope of rights
  • author is copyright owner, joint authorship
    • alternatives – work for hire, transfer of copyright, divisibility of copyrights
  • benefits of agreement – clarification of rights, sharing of rights, avoid copyright as monolith
• good agreements
  • identification of author and third-party materials
  • confirm status – whose name, public domain, creative commons?
  • license to publisher – why not transfer (what does publisher need? license, etc?)
    • rights retained by author – into repository, obligations of open access, future scholarship and teaching

Open Access and Author Control of Copyright (Michael Carroll)

• context of information environment – compare percent of people recycling content
• changes in physical environment require action to avoid threat and instead go to more opportunity
• who’s in charge? the authors – automatic since 1710 for author’s rights
  • standard story is researchers write or impact no for money
    • publisher needs to have an impact, publisher’s brand name becomes a symbol of impact
  • trade copyright for a share in publisher’s trademark – no longer in author’s interest
• scholarly communication – terms of trade fair are not fair
  • ‘unethical’ to not read agreement and just sign
• new methods and output forums – “open access”
  • internet now allows dissemination – material available for free on the internet == “open access”
  • not a specific model that embodies this – just allow people to access a file
    • delayed access – those with membership get it first, eventually all get it; sustainable because inside wall need to be on cutting edge
  • five audiences: serendipitous reader (not searching for specific), under-resourced readers (can’t afford inside costs), interdisciplinary (marginal discipline readers), international (no longer US centric), machine readers (science information is blocked/limited)
• what’s there ? OA Journals
  • fee based OA journals – assumed to be only kind, but a minority
  • PLoS – public library of science is an example; humanities ([[philsophersinprint.org]])

• need to read the copyrights and make open access a possibility

  Contributed Discussion

• Acronyms
  • OA – Open access publication
  • PLOS – public library of science
• Interesting links
  • MIT’s open courseware on facebook
  • Columbia’s Free Culture group also here for the facebook-timid
  • price tag illustration of the cost of academic journals
• Event links
  • Twitter of event – what? seems extreme, but interesting ploy, 160 characters at a time!
  • video links to this event and others at CU

Also, let me know if you’re interested in having your own memoirs that can be submitted via the web/text message.

Here’s the first example, perhaps as a true memoir…

“intrigued, investigating, non-stop. knowledge and life.”