The Algorithm Design Manual by Steven S. Skiena

• There are three desirable properties for a good algorithm. We seek algorithms that are correct and efficient, while being easy to implement. These goals may not be simultaneously achievable.
• There is a fundamental difference between algorithms, which always produce a correct result, and heuristics, which may usually do a good job, but without providing any guarantee.
• Reasonable-looking algorithms can easily be incorrect. Algorithm correctness is a property that must be carefully demonstrated.
• The heart of any algorithm is an idea. If your idea is not clearly revealed when you express an algorithm, then you are using too low-level a notation to describe it.
• It is impossible to prove the correctness of an algorithm for a fuzzily-stated problem. Put another way, ask the wrong problem and you will get the wrong answer.
• An important and honorable technique in algorithm design is to narrow the set of allowable instances until there is a correct and efficient algorithm.
• The best way to prove that an algorithm is incorrect is to produce an instance in which it yields an incorrect answer. Such instances are called counter-examples.
• Hunting for counter-examples is a skill worth developing. It bears some similarity to the task of developing test sets for computer programs, but relies more on inspiration then exhaustion.
• Searching for counterexamples is the best way to disprove the correctness of a heuristic.
• Mathematical induction is usually the right way to verify the correctness of a recursive or incremental insertion algorithm.
• Modeling is the art of formulating your application in terms of precisely described, well-understood problems. Proper modeling is the key to applying algorithmic design techniques to real-world problems. Indeed, proper modeling can eliminate the need to design or even implement algorithms, by relating your application to what has been done before.
• Modeling is only the first step in designing an algorithm for a problem.
• Modeling your application in terms of well-defined structures and algorithms is the most important single step towards a solution.
• Learning to think recursively is learning to look for big things that are made from smaller things of exactly the same type as the big thing.
• Recursive structures occur everywhere in the algorithmic world.
• Algorithms are the most important and durable part of computer science because they can be studied in a language and machine independent way.
• Algorithms can be understood and studied in a language and machine independent manner.
• The worst-case complexity of the algorithm is the function defined by the maximum number of steps taken in any instance of size n.
• The best-case complexity of the algorithm is the function defined by the minimum number of steps taken in any instance of size n.
• The average-case complexity of the algorithm, which is the function defined by the average number of steps over all instances of size n.
• The Big-Oh notation and worst-case analysis are tools that greatly simplify our ability to compare the efficiency of algorithms.
• Although esoteric functions arise in advanced algorithm analysis, a small variety of time complexities suffice and account for most algorithms that are widely used in practice.
• A basic rule of thumb in Big-Oh analysis is that worst-case running time follows from multiplying the largest number of times each nested loop can iterate.
• Pattern matching is the most fundamental algorithmic operation on text strings.
• A logarithm is simply an inverse exponential function.
• Exponential functions grow at a distressingly fast rate. Thus, inverse exponential functions--i.e. Logarithms--grow refreshingly slowly.
• Logarithms arise in any process where things are repeatedly halved.
• Binary search is one of the most powerful ideas in algorithm design.
• Logarithms arise whenever things are repeatedly halved or doubled.
• The maximum benefit from good date structures results from designing your program around them in the first place.
• Data structures can be neatly classified as either continuous or linked, depending upon whether they are based on arrays or pointers.
• Contiguously-allocated structures are composed of single slabs of memory, and include arrays, matrices, heaps, and hash tables.
• Linked data structures are composed of distinct chunks of memory bound together by pointers, and include lists, trees, and graph adjacency lists.
• The array is the fundamental contiguously-allocated data structure. Arrays are structures of fixed-sized data records such that each element can be efficiently located by its index or address.
• Physical continuity between successive data accesses helps exploit the high-speed cache memory on modern computer architectures.
• Pointers are the connections that hold the pieces of linked structures together.
• Pointers represent the address of a location in memory.
• A variable storing a pointer to a given data item can provide more freedom than storing a copy of the item itself.
• Dynamic memory allocation provides us with flexibility on how and where we use our limited storage resources.
• Stacks are simple to implement and very efficient. For this reason, stacks are probably the right container to use when retrieval order doesn’t matter at all, such as when processing batch jobs.
• Algorithmically, LIFO tends to happen in the course of executing recursive algorithms.
• Queues are somewhat trickier to implement than stacks and thus are more appropriate for appliances where the order is important.
• The dictionary data type permits access to data items by content. You stick an item into a dictionary so you can find it when you need it.
• Data structure design must balance all the different operations it supports. The fastest data structure to support both operations A and B may well not be the fastest structure to support either operation A or B.
• Picking the wrong data structure for the job can be disastrous in terms of performance. Identifying the very best data structure is usually not as critical, because there can be several choices that perform similarly.
• Building algorithms around data structures such as dictionaries and priority queues leads to both clean structure and good performance.
• Greedy heuristics always try to grab the best possible thing first.
• When working with a large enough data set, only linear or near linear algorithms are likely to be fast enough.
• Choosing the right data structure is often the key to getting the time complexity down to this point.
• Using smart heuristics like greedy is likely to significantly improve quality over the naive approach.
• Hash tables are a very practical way to maintain a dictionary. THey exploit the fact that looking an item up in an array takes constant time once you have its index.
• A hash function is a mathematical function that maps keys to integers. We will use the value of our hash function as an index into an array, and store our item at that position.
• Chaining is the easiest approach to collision resolution.
• Pragmatically, a hash table is often the best data structure to maintain a dictionary.
• Strings are sequences of characters where the order of the characters matters.
• Text strings are fundamental to a host of computing applications, from programming language parsing/compilation, to web search engines, to biological sequence analysis.
• The primary data structure for representing strings is an array of characters. This allows us constant-time access to the i-th character of the string.
• The most fundamental operation on text strings is substring search.
• The key idea of hashing is to represent a large object using a single number. The goal is a representation of the large object by an entity that can be manipulated in constant time, such that it is relatively unlikely that two different large objects map to the same value.
• Hashing has a variety of clever applications beyond just speeding up search.
• Suffix trees are amazing data structures.
• Sorting is the basic building block that many other algorithms are built around. By understanding sorting, we obtain an amazing amount of power to solve other problems.
• An important algorithm design technique is to use sorting as a basic building block, because many other problems become easy once a set of items is sorted.
• Never be afraid to spend time sorting, provided you use an efficient sorting routine.
• Sorting lies at the heart of many algorithms. Sorting the data is one of the first things any algorithm designer should try in the quest for efficiency.
• Heaps are a simple and elegant data structure for efficiently supporting the priority queue operations insert and extract-min.
• Although other algorithms prove slightly faster in practice, you won’t go wrong using heapsort for sorting data that sites in the computer’s main memory.
• Recursive algorithms reduce large problems into smaller ones.
• Randomization is a powerful tool to improve algorithms with bad worst-case but good average-case complexity. It can be used to make algorithms more robust to boundary cases and more efficient on highly structured input stances that confound heuristic decisions.
• The fundamental step in quicksort is partitioning elements around a pivot.
• Sorting can be used to illustrate most algorithm design paradigms. Data structure techniques, divide-and-conquer, randomization, and incremental construction all lead to efficient sorting algorithms.
• Binary search is a fast algorithm for searching in a sorted array of keys.
• Binary search and its variants are the quintessential divide-and-conquer algorithms.
• One of the most powerful techniques for solving problems is to break them down into smaller, more easily solved pieces. Smaller problems are less overwhelming, and they permit us to focus on details that are lost when we are studying the entire problem.
• Effective parallel processing requires decomposing jobs into at least as many tasks as processors, and is becoming more important with the advent of cluster computing and multicore processors.
• The key to solving many algorithmic problems is to think of them in terms of graphs. Graph theory provides a language for talking about the properties of relationships, and it is amazing how often messy applied problems have a simple description and solution in terms of classical graph properties.
• Designing truly novel graph algorithms is a very difficult task. The key to using graph algorithms effectively in applications lies in correctly modeling your problem so you can take advantage of existing algorithms.
• Graphs can be used to model a wide variety of structures and relationships. Graph-theoretic terminology gives us a language to talk about them.
• Adjacency lists are the right data structure for most applications of graphs.
• The take-home lesson is that even elementary problems like initializing data structures can prove to be bottlenecks in algorithm development. Most programs working with large amounts of data have to run in linear or almost linear time.
• Breadth-first and depth-first searches provide mechanisms to visit each edge and vertex of the graph. They prove the basis of most simple, efficient graph algorithms.
• By storing the vertices in a first-in, first-out (FIFO) queue, we explore the oldest unexplored vertices first.
• By storing the vertices in a last-in, first-out (LIFO) stack, we explore the vertices by lurching along a path, visiting a new neighbor if one is available, backtracking up only when we are surrounded by previously discovered vertices.
• DFS organizes vertices by entry/exit times, and edges into tree and back edges. This organization iw what gives DFS its real power.
• As algorithm design paradigms go, a depth-first search isn’t particularly intimidating. It is surprisingly subtle, however meaning that its correctness requires getting details right.
• Greedy algorithms make the decision of what to do next by selecting the best local option from all available choices without regard to the global structure.
• Most applications of graphs can be reduced to standard grph properties where well-known algorithms can be used. These include minimum spanning tees, shortest paths, and other problems presented in the catalog.
• The maximum flow from s to t always equals the weight of the minimum s-t cut. Thus, flow algorithms can be used to solve general edge and vertex connectivity problems in graphs.
• Proper modeling is the key to making effective use of graph algorithms.
• Designing novel graph algorithms is very hard, so don’t do it. Instead, try to design graphs that enable you to use classical algorithms to model your problem.
• Backtracking is a systematic way to iterate through all the possible configurations of a search space.
• Combinatorial searches, when augmented with tree pruning techniques, can be used to find the optimal solution of small optimization problems. How small depends upon the specific problem, but typical size limits are somewhere between 15 <= N <= 50 items.
• Successful pruning requires looking ahead to see when a solution is doomed to go nowhere, and backing off as soon as possible.
• Even simple pruning strategies can suffice to reduce running time from impossible to instantaneous.
• Clever pruning can make short work of surprisingly hard combinatorial search problems. Proper pruning will have a greater impact on search time than any other factor.
• The simplest method to search in a solution space uses random sampling. It is also called the Monte Carlo method. We repeatedly construct random solutions and evaluate them, stopping as soon as we get a good enough solution, or (more likely) when we are tired of waiting. We report the best solution found over the course of our sampling.
• Simulated annealing is effective because it spends much more of its time working on good elements of the solution space than on bad ones, and because it avoids getting trapped repeatedly in the same local optima.
• Simulated annealing is a simple but effective technique for efficiently obtaining good but not optimal solutions to combinatorial search problems.
• I have never encountered any problem where genetic algorithms seemed to me the right way to attack it. Further, I have never seen any computational results reported using genetic algorithms that have favorably impressed me. Stick to simulated annealing for your heuristic search voodoo needs.
• Your time spent parallelizing code might be better spent enhancing the sequential version.
• I recommend considering parallel processing only after attempts at solving a problem sequentially prove too slow. Even then, I would restrict attention to algorithms that parallelize the problem by partitioning the input into distinct tasks where no communication is needed between the processors, except to collect the final results.
• Once you understand it, dynamic programming is probably the easiest algorithm design technique to apply in practice.
• Dynamic programming is a technique for efficiently implementing a recursive algorithm by storing partial results. The trick is seeing whether the naive recursive algorithm computes the same subproblems over and over and over again. If so, storing the answer for each subproblems in a table to look up instead of recompute can lead to an efficient algorithm.
• Start with a recursive algorithm or definition. Only once we have a correct recursive algorithm do we worry about speeding it up by using a results matrix.
• Dynamic programming is generally the right method for optimization problems on combinatorial objects that have an inherent left to right order among components.
• Dynamic programming is essentially a tradeoff of space for time. Repeatedly recomputing a given quantity is harmless unless the time spent doing so becomes a drag on performance. Then we are better off storing the results of the initial computation and looking them up instead of recomputing them again.
• Explicit caching of the results of recursive calls provides most of the benefits of dynamic programming, including usually the same running time as the more elegant full solution. If you prefer doing extra programming to more subtle thinking, you can stop here.
• Once you understand dynamic programing, it can be easier to work out such algorithms from scratch than to try to look them up.
• For any optimization problem on left-to-right objects, such as characters in a string, elements of a permutation, points around a polygon, or leaves in a search tree, dynamic programming likely leads to an efficient algorithm to find the optimal solution.
• Dynamic programming algorithms are only as correct as the recurrence relations they are based on.
• The running time of any dynamic programming algorithm is a function of two things: (1) number of partial solutions we must keep track of, and (2) how long it will take to evaluate each partial solution.
• Without an inherent left-to-right ordering on the objects, dynamic programming is usually doomed to require exponential space and time.
• The global optimum is often noticeably better than the solution found by typical heuristics. How important this improvement is depends on your application, but it can never hurt.
• Reductions are a way to show that two problems are essentially identical. A fast algorithm for one of the problems implies a fast algorithm for the other.
• A small set of NP-complete problems suffice to prove the hardness of most other hard problems.
• Approximation algorithms guarantee answers that are always close to the optimal solution. They can provide a practical approach to dealing with NP-complete problems.
• The key to algorithm design (or any other problem-solving task) is to proceed by asking yourself questions to guide your thought process.
• Should you get stuck on the problem, the best thing to do is move onto the next question.
• By clearly articulating your reasoning as to why something doesn’t work, you can check whether you have glossed over a possibility that you didn’t want to think hard enough about. It is amazing how often the reason you can’t find a convincing explanation for something is because your conclusion is wrong.
• The distinction between strategy and tactics is important to keep aware of during any design process. Strategy represents the quest for the big picture--the framework around which we construct our path to the goal. Tactics are used to win the minor battles we must fight along the way.
• In problem-solving, it is important to check repeatedly whether you are thinking on the right level.
• Problem-solving is not a science, but part art and part skill. It is one of the skills most worth developing.
• Data structures are not so much algorithms as they are the fundamental constructs around which you build your application. Becoming fluent in what the standard data structures can do for you is essential to get full value from them.
• The abstract data type “dictionary” is one of the most important structures in computer science.
• In practice, it is more important to avoid using a bad data structure than to identify the single best option available.
• For applications involving a moderate-to-large number of keys (say between 100 and 10,000,000), a hash table is probably the right way to go.
• Balanced search trees use local rotation operations to restructure search trees, moving more distant nodes closer to the root while maintaining the in-order search structure of the tree.
• Which tree is best for your application? Probably the one of which you have the best implementation. The flavor of balanced tree is probably not as important as the skill of the programmer who coded it.
• Once a data structure has to be stored outside of main memory, the search time grows by several orders of magnitude.
• The idea behind a B-tree is to collapse several levels of a binary search tree into a single large node, so that we can make the equivalent of several search steps before another disk access is needed.
• Priority queue are useful data structures in simulations, particularly for maintaining a set of future events ordered by time. The are called “priority” queue because they enable you to retrieve items not by the insertion time (as in a stack or queue), nor by a key match (as in a dictionary), but by which item has the highest priority of retrieval.
• Suffix trees and arrays are phenomenally useful data structures for solving string problems elegantly and efficiently. Proper use of suffix trees oten speeds up string processing algorithms [...] to linear time--likely the answer.
• Tries are very simple to build (repeatedly insert new strings) and very fast to search, although they can be expensive in terms of memory.
• Special efforts must be taken to represent very large graphs efficiently.
• The bottom line is that you should try to avoid working in high-dimensional spaces, perhaps by discarding (or projecting away) the least important dimensions.
• Numerical algorithms typically perform repeated floating-point computations, which accumulate error at each operation until, eventually, the results are meaningless.
• A simple and dependable way to test for round-off errors in numerical programs is to run them both at single and double precision, and then think hard whenever there is a disagreement.
• Linear programming is the most important problem in mathematical optimization and operations research.
• The simplex method is the standard algorithm for linear programming. Each constraint in a linear programming problem acts like a knife that carves away a region from the space of possible solutions. We seek the point within the remaining region that maximizes (or minimizes) f(X).
• The bottom line on linear programming is this: you are much better off using an existing LP code than writing your own.
• Linear programming is an algorithmic problem of such economic important that commercial implementations are far superior to free versions.
• It is fundamentally impossible to produce truly random numbers on any deterministic device.
• The best we can hope for are pseudorandom numbers, a stream of numbers that appear as if they were generated randomly.
• The standard algorithm of choice [for generating random numbers] is the linear congruential generator. It is fast, simple, and (if instantiated with the right constraints) gives reasonable pseudorandom numbers.
• Note that the stream of numbers produced by a linear congruential generator repeats the instant the first number repeats.
• High but not arbitrary-precision arithmetic can be conveniently performed using Chinese remainder theorem and modular arithmetic.
• Taking the Fourier transform of a function is equivalent to representing it as the sum of sine functions.
• By eliminating undesirable high and or low frequency components and taking an inverse fourier transform to get us back into the time domain, we can filter an image to remove noise and other artifacts.
• A convolution is the pairwise product of elements from two different sequences, such as in multiplying two n-variable polynomials f and g or comparing two character strings.
• Indeed, “when in doubt, sort” is one of the first rules of algorithm design.
• Degeneracy refers to annoying special cases that must be treated in substantially different ways, such as when two lines intersect in more or less than a single point.
• There are three primary approaches to dealing with degeneracy:
• Ignore it--Make as an operating assumption that your program will work correctly only if no three points are collinear, no three lines meet at a point, no intersections happen at the endpoints of line segments, etc.
• Fake it-- Randomly or symbolically perturb your data so that it becomes nondegenerate.
• Deal with it--Geometric applications can be made more robust by writing special code to handle each of the special cases that arise.
• Geometric computations often involve floating-point arithmetic, which leads to problems with overflows and numerical precision. There are three basic approaches to the issue of numerical stability:
• Integer arithmetic--By forcing all points of interest to lie on a fixed-size integer grid, you can perform exact comparison to test whether any two points are equal or two line segments interest.
• Double precision reals--By using double-precision floating point numbers, you may get lucky and avoid numerical errors.
• Arbitrary precision arithmetic--This is certain to be correct, but also to be slow.
• Sets are collections of symbols whose order is assumed to carry no significant, while strings are defined by the sequence or arrangement of symbols.
• The assumption of a fixed order makes it possible to solve string problems much more efficiently than set problems, through techniques such as dynamic programming and advanced data structures like suffix trees.
• A good algorithm designer does not reinvent the wheel, and a good programmer does not rewrite code that other people have written. Picasso put it best: “Good artists borrow. Great artists steal.”

Finding Your Funny Bone! by Nancy Gold

• Physical theatre training is perhaps the most difficult training to capture within the pages of a book.
• The comedian or clown has to find his or her stupid and stay in it for the audience to be reminded of their humanity. And it takes a great deal of courage, intelligence, and grace to stay stupid.
• All of ART--no matter what kind it is--is an exchange of ENERGY between the audience and the artwork or performer, between the performers on the stage and between you as the performer and the character.
• Now just like a painter uses oil or watercolour or pastel or a sculptor uses clay or steel or metal, you will use the Air Around You. It is your medium.
• If you do not receive what is given, then the communication is blocked. The full emotional experience is not being felt or the person receiving is not fully receiving your communication. He is either not open to receiving the energy, or he is not seeing the amount of energy coming, or he is scared to receive it, or he wants to control the energy flow and transform the powerpoint of the energy to his definition of how much energy to give and receive.
• The ultimate comedy brings tears to your eyes. Laughing is the other side of crying. So if you can laugh, you can cry.
• The essence of Neutral is to be just there in space. It is not happy or sad, it is just there.
• Body shape is really important in creating a character. It is essential in creating a comedy character. The body shape creates the Walk and emotional Talk of your character.
• You need the audience to see what you are doing. And you need to include them in everything you do. So, being conscious of staging is an important element of mime, which is also helpful in comed.
• WHen you are making an object for people to see, it is most powerful in a direct line to the audience.
• Real feelings and reactions are funny. Trying to be funny and/or “showing” feelings are formulas for a flop.
• A clown nose is one of the smallest masks you can wear.
• A key to comedy is having a problem and a deep need to fix it.
• It is important to push the envelope in the exploration and discovery. The most interesting work and discover usually begin when the group has decided they have had enough and it is time to stop. This is when they lose themselves in the concept of gorilla and become the essence of the animal.

EGO IS THE ENEMY by Ryan Holiday

• We must begin by seeing ourselves and the world in a new way for the first time. Then we must fight to be different and fight to stay different—that’s the hard part.
• Wherever you are, whatever you’re doing, your worst enemy already lives inside you: your ego.
• The need to be better than, more than, recognized for, far past any reasonable utility—that’s ego. It’s the sense of superiority and certainty that exceeds the bounds of confidence and talent.
• Most of us aren’t “egomaniacs,” but ego is there at the root of almost every conceivable problem and obstacle, from why we can’t win to why we need to win all the time and at the expense of others.
• With every ambition and goal we have—big or small—ego is there undermining us on the very journey we’ve put everything into pursuing.
• Without an accurate accounting of our own abilities compared to others, what we have is not confidence but delusion.
• Just one thing keeps ego around—comfort. Pursuing great work—whether it is in sports or art or business—is often terrifying. Ego soothes that fear. It’s a salve to that insecurity. Replacing the rational and aware parts of our psyche with bluster and self-absorption, ego tells us what we want to hear, when we want to hear it.
• At any given time in life, people find themselves at one of three stages. We’re aspiring to something—trying to make a dent in the universe. We have achieved success—perhaps a little, perhaps a lot. Or we have failed—recently or continually. Most of us are in these stages in a fluid sense—we’re aspiring until we succeed, we succeed until we fail or until we aspire to more, and after we fail we can begin to aspire or succeed again.
• We can seek to rationalize the worst behavior by pointing to outliers. But no one is truly successful because they are delusional, self-absorbed, or disconnected.
• Among men who rise to fame and leadership two types are recognizable—those who are born with a belief in themselves and those in whom it is a slow growth dependent on actual achievement.
• One must ask: if your belief in yourself is not dependent on actual achievement, then what is it dependent on? The answer, too often when we are just setting out, is nothing. Ego. And this is why we so often see precipitous rises followed by calamitous falls.
• One might say that the ability to evaluate one’s own ability is the most important skill of all. Without it, improvement is impossible.
• own work. Any and every narcissist can do that. What is rare is not raw talent, skill, or even confidence, but humility, diligence, and self-awareness.
• It’s a temptation that exists for everyone—for talk and hype to replace action.
• Writing, like so many creative acts, is hard.
• In actuality, silence is strength—particularly early on in any journey.
• Anyone can talk about himself or herself. Even a child knows how to gossip and chatter. Most people are decent at hype and sales. So what is scarce and rare? Silence. The ability to deliberately keep yourself out of the conversation and subsist without its validation. Silence is the respite of the confident and the strong.
• Talk depletes us. Talking and doing fight for the same resources. Research shows that while goal visualization is important, after a certain point our mind begins to confuse it with actual progress. The same goes for verbalization.
• After spending so much time thinking, explaining, and talking about a task, we start to feel that we’ve gotten closer to achieving it.
• Success requires a full 100 percent of our effort, and talk flitters part of that effort away before we can use it.
• Doing great work is a struggle.
• The only relationship between work and chatter is that one kills the other.
• Appearances are deceiving. Having authority is not the same as being an authority.
• Impressing people is utterly different from being truly impressive.
• What you choose to do with your time and what you choose to do for money works on you.
• The power of being a student is not just that it is an extended period of instruction, it also places the ego and ambition in someone else’s hands.
• The pretense of knowledge is our most dangerous vice, because it prevents us from getting any better.
• The mixed martial arts pioneer and multi-title champion Frank Shamrock has a system he trains fighters in that he calls plus, minus, and equal. Each fighter, to become great, he said, needs to have someone better that they can learn from, someone lesser who they can teach, and someone equal that they can challenge themselves against.
• “False ideas about yourself destroy you.
• You can’t learn if you think you already know.
• The art of taking feedback is such a crucial skill in life, particularly harsh and critical feedback. We not only need to take this harsh feedback, but actively solicit it, labor to seek out the negative precisely when our friends and family and brain are telling us that we’re doing great.
• To become what we ultimately hope to become often takes long periods of obscurity, of sitting and wrestling with some topic or paradox.
• In our endeavors, we will face complex problems, often in situations we’ve never faced before. Opportunities are not usually deep, virgin pools that require courage and boldness to dive into, but instead are obscured, dusted over, blocked by various forms of resistance. What is really called for in these circumstances is clarity, deliberateness, and methodological determination.
• Passion typically masks a weakness.
• What humans require in our ascent is purpose and realism. Purpose, you could say, is like passion with boundaries. Realism is detachment and perspective.
• Purpose is about pursuing something outside yourself as opposed to pleasuring yourself.
• Passion is form over function. Purpose is function, function, function.
• It’d be far better if you were intimidated by what lies ahead—humbled by its magnitude and determined to see it through regardless.
• Clear the path for the people above you and you will eventually create a path for yourself.
• When you are just starting out, we can be sure of a few fundamental realities: 1) You’re not nearly as good or as important as you think you are; 2) You have an attitude that needs to be readjusted; 3) Most of what you think you know or most of what you learned in books or in school is out of date or wrong.
• attach yourself to people and organizations who are already successful and subsume your identity into theirs and move both forward simultaneously. It’s certainly more glamorous to pursue your own glory—though hardly as effective.
• Greatness comes from humble beginnings; it comes from grunt work. It means you’re the least important person in the room—until you change that with results.
• There is an old saying, “Say little, do much.” What we really ought to do is update and apply a version of that to our early approach. Be lesser, do more.
• Find what nobody else wants to do and do it.
• Produce more than everyone else and give your ideas away
• It doesn’t matter how talented you are, how great your connections are, how much money you have. When you want to do something—something big and important and meaningful—you will be subjected to treatment ranging from indifference to outright sabotage. Count on it.
• Those who have subdued their ego understand that it doesn’t degrade you when others treat you poorly; it degrades them.
• Restraint is a difficult skill but a critical one. You will often be tempted, you will probably even be overcome. No one is perfect with it, but try we must.
• It is a timeless fact of life that the up-and-coming must endure the abuses of the entrenched.
• you’re not able to change the system until after you’ve made it.
• We tend to think that ego equals confidence, which is what we need to be in charge. In fact, it can have the opposite effect.
• It is natural for any young, ambitious person (or simply someone whose ambition is young) to get excited and swept up by their thoughts and feelings.
• The more creative we are, the easier it is to lose the thread that guides us.
• Our imagination—in many senses an asset—is dangerous when it runs wild. We have to rein our perceptions in. Otherwise, lost in the excitement, how can we accurately predict the future or interpret events?
• Living clearly and presently takes courage. Don’t live in the haze of the abstract, live with the tangible and real, even if—especially if—it’s uncomfortable. Be part of what’s going on around you. Feast on it, adjust for it.
• There’s no one to perform for. There is just work to be done and lessons to be learned, in all that is around us.
• Pride blunts the very instrument we need to own in order to succeed: our mind. Our ability to learn, to adapt, to be flexible, to build relationships, all of this is dulled by pride.
• Pride takes a minor accomplishment and makes it feel like a major one.
• Punching above your weight is how you get injured. Pride goeth before the fall.
• If you’re doing the work and putting in the time, you won’t need to cheat, you won’t need to overcompensate.
• Receive feedback, maintain hunger, and chart a proper course in life.
• We must prepare for pride and kill it early—or it will kill what we aspire to.
• We must be on guard against that wild self-confidence and self-obsession.
• Privately thinking you’re better than others is still pride. It’s still dangerous.
• “Don’t boast about what hasn’t happened yet.”
• The best plan is only good intentions unless it degenerates into work.
• “You can’t build a reputation on what you’re going to do,”
• Where we decide to put our energy decides what we’ll ultimately accomplish.
• You can lie to yourself, saying that you put in the time, or pretend that you’re working, but eventually someone will show up. You’ll be tested. And quite possibly, found out.
• Make it so you don’t have to fake it—that’s the key.
• Every time you sit down to work, remind yourself: I am delaying gratification by doing this. I am passing the marshmallow test. I am earning what my ambition burns for. I am making an investment in myself instead of in my ego.
• Work is pushing through the pain and crappy first drafts and prototypes.
• Work doesn’t want to be good. It is made so, despite the headwind.
• Of course, what is truly ambitious is to face life and proceed with quiet confidence in spite of the distractions.
• Success is intoxicating, yet to sustain it requires sobriety. We can’t keep learning if we think we already know everything. We cannot buy into myths we make ourselves, or the noise and chatter of the outside world. We must understand that we are a small part of an interconnected universe. On top of all this, we have to build an organization and a system around what we do—one that is about the work and not about us.
• It takes a special kind of humility to grasp that you know less, even as you know and grasp more and more.
• With accomplishment comes a growing pressure to pretend that we know more than we do. To pretend we already know everything.
• No matter what you’ve done up to this point, you better still be a student. If you’re not still learning, you’re already dying.
• Learn from everyone and everything. From the people you beat, and the people who beat you, from the people you dislike, even from your supposed enemies.
• Too often, convinced of our own intelligence, we stay in a comfort zone that ensures that we never feel stupid (and are never challenged to learn or reconsider what we know). It obscures from view various weaknesses in our understanding, until eventually it’s too late to change course. This is where the silent toll is taken.
• The solution is as straightforward as it is initially uncomfortable: Pick up a book on a topic you know next to nothing about. Put yourself in rooms where you’re the least knowledgeable person. That uncomfortable feeling, that defensiveness that you feel when your most deeply held assumptions are challenged—what about subjecting yourself to it deliberately? Change your mind. Change your surroundings.
• Crafting stories out of past events is a very human impulse. It’s also dangerous and untrue. Writing our own narrative leads to arrogance.
• once you win, everyone is gunning for you.
• Facts are better than stories and image.
• “The way to do really big things seems to be to start with deceptively small things.”
• Instead of pretending that we are living some great story, we must remain focused on the execution—and on executing with excellence.
• That’s how it seems to go: we’re never happy with what we have, we want what others have too. We want to have more than everyone else. We start out knowing what is important to us, but once we’ve achieved it, we lose sight of our priorities. Ego sways us, and can ruin us.
• All of us waste precious life doing things we don’t like, to prove ourselves to people we don’t respect, and to get things we don’t want.
• Ego leads to envy and it rots the bones of people big and small.
• The farther you travel down that path of accomplishment, whatever it may be, the more often you meet other successful people who make you feel insignificant.
• Let’s be clear: competitiveness is an important force in life. It’s what drives the market and is behind some of mankind’s most impressive accomplishments. On an individual level, however, it’s absolutely critical that you know who you’re competing with and why, that you have a clear sense of the space you’re in.
• It’s time to sit down and think about what’s truly important to you and then take steps to forsake the rest. Without this, success will not be pleasurable, or nearly as complete as it could be. Or worse, it won’t last.
• The more you have and do, the harder maintaining fidelity to your purpose will be, but the more critically you will need to.
• Find out why you’re after what you’re after. Ignore those who mess with your pace. Let them covet what you have, not the other way around. Because that’s independence.
• With success, particularly power, come some of the greatest and most dangerous delusions: entitlement, control, and paranoia.
• Ego is its own worst enemy. It hurts the ones we love too.
• A smart man or woman must regularly remind themselves of the limits of their power and reach.
• When we’re aspiring or small time, we can be idiosyncratic, we can compensate for disorganization with hard work and a little luck. That’s not going to cut it in the majors. In fact, it’ll sink you if you can’t grow up and organize.
• It turns out that becoming a great leader is difficult. Who knew?!
• Management? That’s the reward for all your creativity and new ideas? Becoming the Man? Yes—in the end, we all face becoming the adult supervision we originally rebelled against. Yet often we react petulantly and prefer to think: Now that I’m in charge, things are going to be different!
• As you become successful in your own field, your responsibilities may begin to change. Days become less and less about doing and more and more about making decisions. Such is the nature of leadership. This transition requires reevaluating and updating your identity. It requires a certain humility to put aside some of the more enjoyable or satisfying parts of your previous job. It means accepting that others might be more qualified or specialized in areas in which you considered yourself competent—or at least their time is better spent on them than yours.
• Sometimes systems are better decentralized. Sometimes they are better in a strict hierarchy. Every project and goal deserves an approach fitted perfectly to what needs to be done.
• What matters is that you learn how to manage yourself and others, before your industry eats you alive.
• Ego needs honors in order to be validated. Confidence, on the other hand, is able to wait and focus on the task at hand regardless of external recognition.
• This is one of the most dangerous ironies of success—it can make us someone we never wanted to be in the first place.
• Early in your career, you’ll notice that you jump on every opportunity to do so. As you become more accomplished, you’ll realize that so much of it is a distraction from your work—time spent with reporters, with awards, and with marketing are time away from what you really care about.
• It doesn’t make you a bad person to want to be remembered. To want to make it to the top. To provide for yourself and your family. After all, that’s all part of the allure.
• Play for the name on the front of the jersey, he says, and they’ll remember the name on the back.
• Ego tells us that meaning comes from activity, that being the center of attention is the only way to matter.
• When we lack a connection to anything larger or bigger than us, it’s like a piece of our soul is gone.
• Creativity is a matter of receptiveness and recognition. This cannot happen if you’re convinced the world revolves around you.
• Once we’ve made it, we tend to think that ego and energy is the only way to stay there. It’s not.
• We have to fight to stay sober, despite the many different forces swirling around our ego.
• But as Merkel supposedly said, “You can’t solve . . . tasks with charisma.”
• Sobriety is the counterweight that must balance out success. Especially if things keep getting better and better.
• Most successful people are people you’ve never heard of. They want it that way.
• Endless ambition is easy; anyone can put their foot down hard on the gas. Complacency is easy too; it’s just a matter of taking that foot off the gas.
• The world conspires against us in many ways, and the laws of nature say that everything regresses toward the mean.
• Just because you did something once, doesn’t mean you’ll be able to do it successfully forever.
• Reversals and regressions are as much a part of the cycle of life as anything else.
• No one is permanently successful, and not everyone finds success on the first attempt.
• Almost without exception, this is what life does: it takes our plans and dashes them to pieces. Sometimes once, sometimes lots of times.
• If ego is often just a nasty side effect of great success, it can be fatal during failure.
• According to Greene, there are two types of time in our lives: dead time, when people are passive and waiting, and alive time, when people are learning and acting and utilizing every second.
• Lacking the ability to examine ourselves, we reinvest our energy into exactly the patterns of behavior that caused our problems to begin with.
• In life, there will be times when we do everything right, perhaps even perfectly. Yet the results will somehow be negative: failure, disrespect, jealousy, or even a resounding yawn from the world.
• “Success is peace of mind, which is a direct result of self-satisfaction in knowing you made the effort to do your best to become the best that you are capable of becoming.”
• There are many ways to hit bottom. Almost everyone does in their own way, at some point.
• The bigger the ego the harder the fall.
• The world can show you the truth, but no one can force you to accept it.
• People make mistakes all the time.
• Ego kills what we love. Sometimes, it comes close to killing us too.
• Most trouble is temporary . . . unless you make that not so.
• Only ego thinks embarrassment or failure are more than what they are. History is full of people who suffered abject humiliations yet recovered to have long and impressive careers.
• “He who fears death will never do anything worthy of a living man,” Seneca once said. Alter that: He who will do anything to avoid failure will almost certainly do something worthy of a failure.
• The only real failure is abandoning your principles.
• If your reputation can’t absorb a few blows, it wasn’t worth anything in the first place.
• This is characteristic of how great people think. It’s not that they find failure in every success. They just hold themselves to a standard that exceeds what society might consider to be objective success. Because of that, they don’t much care what other people think; they care whether they meet their own standards. And these standards are much, much higher than everyone else’s.
• Your potential, the absolute best you’re capable of—that’s the metric to measure yourself against.
• The more successful or powerful we are, the more there will be that we think we need to protect in terms of our legacy, image, and influence. If we’re not careful, however, we can end up wasting an incredible amount of time trying to keep the world from displeasing or disrespecting us.
• Yet we find that what defines great leaders like Douglass is that instead of hating their enemies, they feel a sort of pity and empathy for them.
• At various points in our lives, we seem to have different capacities for forgiveness and understanding.
• There is no way around it: We will experience difficulty. We will feel the touch of failure.
• “People learn from their failures. Seldom do they learn anything from success.”
• Most of us can’t handle uncomfortable self-examination.
• We all experience success and failure in our own way.

THE GERVAIS PRINCIPLE by Venkatesh Rao

• This is just a cheap and easily digestible basic organizational literacy 101 guide written in what is hopefully an accessible and contemporary style rather than impenetrable Nietzschean verse.
• But there is a cost to getting organizationally literate. This ability, once acquired, cannot be un-acquired.
• Literacy of any sort gives you the power to recognize and unambiguously label things that the illiterate can easily ignore as noise, fads and bullshit.
• Literacy of any sort is a good thing. Organizational literacy is liberal education in the best sense of the term: it increases your freedom by making you more alive to the possibilities around you.
• The Office is not a random series of cynical gags aimed at momentarily alleviating the existential despair of low-level grunts. It is a fully realized theory of management that falsifies 83.8% of the business section of the bookstore.
• The theory begins with Hugh MacLeod’s well-known cartoon, Company Hierarchy, shown below, and its cornerstone is something I will call “The Gervais Principle,” which supersedes both the Peter Principle and its successor, The Dilbert Principle.
• Hugh MacLeod’s cartoon is a pitch-perfect symbol of an unorthodox school of management based on the axiom that organizations don’t suffer pathologies; they are intrinsically pathological constructs.
• The Sociopath (capitalized) layer comprises the Darwinian/Protestant Ethic will-to-power types who drive an organization to function despite itself.
• The Clueless layer is what Whyte called the “Organization Man,” but the archetype inhabiting the middle has evolved a good deal since Whyte wrote his book in the fifties.
• The Losers are not social losers (as in the opposite of “cool”), but people who have struck bad bargains economically – giving up capitalist striving for steady paychecks.
• Of all organization men, the true executive is the one who remains most suspicious of The Organization.
• A Sociopath with an idea recruits just enough Losers to kick off the cycle. As it grows, it requires a Clueless layer to turn it into a controlled reaction, rather than a runaway explosion. Eventually, as value hits diminishing returns, both the Sociopaths and Losers make their exits, and the Clueless start to dominate. Finally, the hollow brittle shell collapses on itself, and anything of value is recycled by the Sociopaths, according to meta-firm logic.
• The good news is that Losers have two ways out, which we’ll get to later: turning Sociopath or turning into bare-minimum performers. The Losers destined for Cluelessness do not have a choice.
• The Sociopaths enter and exit organizations at will, at any stage, and do whatever it takes to come out on top. They contribute creativity in early stages of a organization’s life, neurotic leadership in the middle stages, and cold-bloodedness in the later stages, where they drive decisions like mergers, acquisitions and layoffs that others are too scared or too compassionate to drive. They are also the ones capable of equally impersonally exploiting a young idea for growth in the beginning, killing one good idea to concentrate resources on another at maturity, and milking an end-of-life idea through harvest-and-exit market strategies.
• The Losers like to feel good about their lives. They are the happiness seekers, rather than will-to-power players, and enter and exit reactively, in response to the meta-Darwinian trends in the economy. But they have no more loyalty to the firm than the Sociopaths. They do have a loyalty to individual people, and a commitment to finding fulfillment through work when they can, and coasting when they cannot.
• The Clueless are the ones who lack the competence to circulate freely through the economy (unlike Sociopaths and Losers), and build up a perverse sense of loyalty to the firm, even when events make it abundantly clear that the firm is not loyal to them.
• The Gervais Principle is this: Sociopaths, in their own best interests, knowingly promote over-performing Losers into middle-management, groom under-performing Losers into Sociopaths, and leave the average bare-minimum-effort Losers to fend for themselves.
• The Peter Principle states that all people are promoted to the level of their incompetence. It is based on the assumption that future promotions are based on past performance.
• The Peter Principle is wrong for the simple reason that executives aren’t that stupid, and because there isn’t that much room in an upward-narrowing pyramid.
• Scott Adams, seeing a different flaw in The Peter Principle, proposed The Dilbert Principle: that companies tend to systematically promote their least-competent employees to middle management to limit the damage they can do. This again is untrue.
• So why is promoting over-performing Losers logical? The simple reason is that if you over-perform at the Loser level, it is clear that you are an idiot. You’ve already made a bad bargain, and now you’re delivering more value than you need to, making your bargain even worse. Unless you very quickly demonstrate that you know your own value by successfully negotiating more money and/or power, you are marked out as an exploitable clueless Loser.
• A Loser who can be suckered into bad bargains is set to become one of the Clueless.
• The future Sociopath must be an under-performer at the bottom.
• the Loser game is not worth becoming good at.
• The career of the Loser is the easiest to understand. Having made a bad bargain, and not marked for either Clueless or Sociopath trajectories, he or she must make the best of a bad situation. The most rational thing to do is slack off and do the minimum necessary.
• The Sociopaths know that the only way to make an organization capable of survival is to buffer the intense chemistry between the producer-Losers and the leader-Sociopaths with enough Clueless padding in the middle to mitigate the risks of business.
• The average-performing , mostly-disengaged Losers can create diminishing-margins profitability, but not sustainable performance or growth.
• the standard promotion/development path is primarily designed to maneuver the Clueless into position wherever they are needed.
• The Sociopaths must be freed up as much as possible to actually run the business, with or without official titles.
• Sociopaths use Powertalk as a coded language with which to simultaneously sustain the (necessary) delusions of the Clueless and communicate with each other.
• The Gervais Principle operates at the slow tempo of promotions, demotions, layoffs and hirings. The bulk of organizational life, however, plays out much faster: one conversation at a time.
• Powertalk is the in-group language of the Sociopaths.
• Posturetalk is the language spoken by the Clueless to everybody.
• Sociopaths and Losers talk back to the Clueless in a language called Babytalk that seems like Posturetalk to the Clueless.
• Among themselves, Losers speak a language called Gametalk.
• I won’t cover it at all, but you can learn all about it in the pop classics on transactional analysis (TA) from 30 years ago: Eric Berne’s Games People Play and What Do You Say after You Say Hello, and Thomas Harris’ I’m OK–You’re OK.
• Add these three books to the two I already referenced, The Organization Man and Images of Organization.)
• Sociopaths and Losers rarely speak to each other at all.
• Losers can partially understand, but not speak Powertalk. To them, Powertalk is a spectator sport.
• Multiple layers of meaning are not what make Powertalk unique.
• What distinguishes Powertalk is that with every word uttered, the power equation between the two speakers shifts just a little.
• Gametalk leaves power relations unchanged because its entire purpose is to help Losers put themselves and each other into safe pigeonholes that validate do-nothing life scripts.
• In Powertalk, you play with valuable currency, usually reality-information. In the other languages, you are playing with no stakes.
• The most important enabling factor in being able to speak Powertalk is simply the possession of table stakes. Without it, whatever you say is Posturetalk.
• The only Powertalk you can speak without any table stakes is “silence.”
• The bulk of Sociopath communication takes places out in the open, coded in Powertalk, right in the presence of non-Sociopaths.
• So effective Sociopaths stick with steadfast discipline to the letter of the law, internal and external, because the stupidest way to trip yourself up is in the realm of rules where the Clueless and Losers get to be judges and jury members.
• Though distant from our worlds, criminal worlds have the one advantage that they do not need to maintain the fiction that the organization is not pathological, so they are revealing to study.
• People who try earnestly to learn Powertalk from recipe books end up merely expanding their Posturetalk vocabulary.
• Toy Guns is the vocabulary of empty machismo.
• A good way to remember this is to think of Powertalk as decisions about what verbal tactics to use when, and with what. The answer to with what is usually a part of your table-stakes.
• Bottomline: you cannot learn Powertalk from books.
• Vocabulary expansion efforts can at best put the finishing touches on organically acquired language skills.
• There is no shortcut to organic language acquisition; reading well-written stuff and writing constantly are the only way.
• You learn through real Powertalk conversations with other Sociopaths.
• Along every learning curve, between the early, instant gratification (which I’ve been hawking so far) and useful mastery, there is a long hard slog: what Seth Godin calls the “Dip.”
• the depth of any transaction is limited by the depth of the shallower party.
• At the level of abstraction that we are concerned with, all theories of developmental psychology – Freud’s, Piaget’s, Erikson’s, Maslow’s – say roughly the same thing about arrested development: you are born Clueless, and get clued in in fits and starts.
• Well-adjustedness is a measure of the degree to which your worldview is socially acceptable and appropriate in a given environment.
• If your situational reactions are generally appropriate but against your best interests, you are a well-adjusted Loser. If they are both appropriate and in your best interests, you are a Sociopath. If your reactions are inappropriate (whether or not they are in your best interests – sometimes they are), you are Clueless.
• Your development is arrested by your strengths, not your weaknesses. Arrested-development behavior is caused by a strength-based addiction. The mediocre develop faster than either the talented or the untalented.
• A strength in one situation is merely an entrenched piece of arrested development in another.
• The Clueless distort reality. The Losers distort rewards and penalties. The Sociopaths distort the metaphysics of human life.
• it is always hard for a student to teach a teacher, even if the student is studying a subject that is more advanced than the one the teacher teaches.
• To manufacture original thought you have to listen to reality in open ways for data.
• The language of winning and losing and debts is useful for all interactions, but it is only consequential, and capable of causing power shifts, when Sociopaths are involved.
• Social clubs of any sort divide the world into an us and a them. We are better than them. Any prospective new member who could raise the average prestige of a club is, by definition, somebody who is too good for that club.
• Status illegibility is necessary to keep a group of Losers stable.
• forming groups is a Loser activity).
• Loser dynamics are largely driven by Lake-Wobegon-effect snow jobs, which obscure pervasive mediocrity.
• Only the alpha can legitimately confer the #2 title, and there is rarely a good reason for the alpha to do so unless he or she is planning to exit.
• A social skill, such as joke-telling ability, is a behavior whose effectiveness is determined by the reaction of a group. A joke is funny if the audience laughs.
• In general, the creation of social capital depends entirely on the reactions of the audience.
• Laugh/frown votes are a powerful weapon for the passive members of any situational group.
• Sociopath jokes usually involve straight-faced delivery and private laughter, with no hint of mockery.
• Among the Sociopaths, status is irrelevant. Table stakes and skill at using them is what matters. Sociopaths pay attention to what you have, and how well you bargain with it. Not who you are.
• But among Losers, status is real, and it matters.
• Among Losers, in specific situations, status may go up or down, but overall, it just goes round and round. There is no grand status hierarchy. Only a top, a bottom, and an illegible middle.
• happiness is entirely a social phenomenon, and there’s plenty of evidence that the best way (and from my reading, the only way) to get happy is to get sociable.
• The basic mechanism by which Sociopaths transfer blame to the Clueless, while reducing the overall severity of the penalty, is an application of Hanlon’s Razor: never attribute to malice what can be adequately explained by stupidity.
• Because Hanlon’s Razor is often true, it is a believable dodge even when it is not.
• When means are defensible, but ends are not, Sociopaths engineer execution failures via indirection and abstraction in the requests they make, thereby achieving their ends via “lucky accidents.” This is the second kind of Hanlon Dodge.
• when you genuinely want to give reports responsibilities that help them grow, you give them autonomy where they are strong. When you want to use them in engineered “failures” that give you the outcomes you want, you give them autonomy in areas where they are weak.
• The Clueless and Losers debate whether or not ends justify the means. Sociopaths use whatever is justifiable to cover up whatever they want to get done. The result is a theater of justification.
• Loser group dynamics offer a natural exploit: almost anyone can be made to ally with, or turn against, anybody else, with no need to manufacture reasons.
• A successful group systematically overvalues its capabilities and develops a blindness to its weaknesses.
• Losers have a genuine sense of honor. The want to accept fair blame for failures and fair credit for successes.
• You’ve probably heard a piece of cynical wisdom: the purpose of a form is not to serve the person who submits it, but to protect the person who processes it.
• The risk-management work of an organization can be divided into two parts: the unpredictable part that is the responsibility of the line hierarchy, and the predictable, repetitive part that is the responsibility of the staff hierarchy.
• As a friend once remarked, tax law is complex for a reason: its primary purpose is to catalyze the growth of complicated exception-handling on top of an apparently simple percentage calculation.
• There are only three ways to get a bureaucracy to do anything it wasn’t designed to do: by stealth, with secret and deniable support from allies in the staff hierarchy; by getting air-cover from a sufficiently high-up Sociopath who can play poker with whichever oversubscribed Sociopath is in charge of exception-handling for the specific process (i.e. jumping the appeals queue and calling in favors to ensure the required ruling); and through corruption and bribery.
• That is what Sociopaths ultimately do with their lives if they survive long enough: generate amoral power from increasing inner emptiness, transforming themselves into forces of nature.
• When Sociopaths turn their attentions en masse to new frontiers, they leave behind complete cargo cults that continue to function for a while.
• The Sociopath journey begins with what is essentially a religious dissatisfaction. A dissatisfaction that awakens the first time Sociopaths contemplate their situation in life.
• Sociopaths progressively rip away layer after layer of social reality.
• Small minds discuss people, average minds discuss events. Great minds discuss ideas.
• When Sociopaths accept the divine roles that the Clueless and Losers eagerly thrust upon them, they find themselves ruling the realities of others. But any human stand-in for an omnipotent conception of divinity must ultimately betray the believer.
• The key, when betraying the Clueless, is to get them to blame themselves. With Losers, the key is to get them to blame each other.
• The Clueless seek idols to emulate.
• the minds of Losers turn to endlessly reliving social events and the associated churn of status and emotions.
• By operating with a more complete calculus, Sociopaths are able to manipulate this world through the divide-and-conquer mechanisms.
• Guilt is the one emotion that Losers cannot always resolve for themselves, since it sometimes requires quantities of forgiveness that mere humans cannot dispense, but priests can, as reserve bankers of the fiat currencies of Loser emotional life.
• Losers are usually collectively, rather than singly betrayed, but Sociopaths are created one at a time.
• Sociopaths find ideas contending in their minds. The creative destruction they script in the world of Losers and Clueless is mirrored by a creative destruction in their minds. This process creates power, but destroys meaning, especially the meanings of social realities. The result is increasing inner emptiness and external power.
• Recall that Sociopaths create meaning for others through the things they subtract, rather than the things they add.
• This is something conspiracy theorists typically don’t get: manufacturing fake realities is very hard. But subtractive simplification of reality is much easier, and yields just as much power.
• Sociopaths exercise agency on behalf of others. They do not grab power. Power is simply ceded to them.
• Sociopathy is not about ripping off a specific mask from the face of social reality. It is about recognizing that there are no social realities. There are only masks.
• Social realities exist as a hierarchy of increasingly sophisticated and specialized fictions for those predisposed to believe that there is something special about the human condition, which sets our realities apart from the rest of the universe.
• By humanizing the non-human universe, we make the human special.
• All that is required is to control people who believe in fairness, is to remove any evidence suggesting that the world might fundamentally not be a fair place, and mask it appropriately with a justice principle such as an afterlife calculus, or a retirement fantasy.
• When a layer of social reality is penetrated and turned into a means for manipulating the realities of others, it is automatically devalued.
• Once the Sociopath overcomes reality shock and frames his life condition as one defined by an absence of ultimate parental authority, and the fictitious nature of all social realities, he experiences a great sense of unlimited possibilities and power.
• Sociopath freedom of speech is the freedom to bullshit: they are bullshit artists in the truest sense of the phrase.
• Non-Sociopaths, as Jack Nicholson correctly argued, really cannot handle the truth. The truth of an absent god. The truth of social realities as canvases for fiction for those who choose to create them.
• Creative destruction is not a script, but the absence of scripts.
• But freedom can also be a scary condition. It offers no canned reasons to do one thing instead of another, or even do anything at all. It offers no fixed motivations. There is nobody to blame for failures, no meaningful external validation for success.
• If physics allows it, you can do it. The consequences mean whatever you decide they mean.
• What is known cannot now be un-known. There is no way to reverse the effects of the red pill of Sociopathy.
• Peter, Michael and Samir, at the start of the story, are grappling with the gradual draining of freedom that is the consequence of socialization into middle-class scripts. It is a kind of loss, and where there is loss, there will be Kubler-Ross.
• The major theme of Office Space, unlike The Office, is not deciphering and navigating the gridlock on the road to power, but exiting the rat-race altogether, to a state held up as an ideal of freedom: exile.
• The association between criminality and exile is a widely recognized one. This is why the term outlaw has connotations of both exile and criminality.
• While exits from the prevailing social order are not exactly blocked off, a toll must be paid in order to pass through: to even seek an existence outside the legitimate part of the social order is to accept being marked as a potential criminal.
• Not all criminal classes are exile classes, and not all exile classes are criminal classes.
• This is the central cognitive dissonance in Peter’s life: there is literally nothing keeping him trapped in his gridlock script, not even money.
• Tom’s idea of freedom is that holy grail of exit scripts: a passive income stream.
• our perceptions of objective value are colored by our perceptions of class and social status.
• Most people recognize that many blue-collar jobs pay more than white-collar jobs. Few act on that recognition.
• Freedom is about more than walking through an unlocked exit door. It is also about figuring out how to avoid the default catastrophic fates and how to deal with the burden of negative perceptions associated with seeking an exit. It is about surviving exit wounds, if you’ll forgive a terrible pun. The first step is to choose a voluntary exit rather than waiting to be forced out.
• Every month, a fresh cohort of script-bound middle-class white-collar workers attempts to break free, armed with nothing more than some savings, vague startup or artistic dreams, and the idea that there can be more to life than gradual dehumanization to Miltonhood.
• Most only manage empty gestures and remain fundamentally trapped, never even making it to a real exit path. The rockstar road, backpacking walkabout and startup dream all turn out to be improvised subplots within the main script rather than clean breaks from it. As a cynical reader once remarked to me, you can take a person out of the middle class, but you cannot take the middle class out of a person.
• Breaking through internal mental barriers is the essential step.
• You can be free in a cubicle or remain trapped while wandering in the desert.
• The only true exit is to a freer mind.
• Organizational literacy is a skill.
• The equivalent of writing is practicing behaviors designed to influence people and organizations.

Gray Hat Hacking

• It has been proven over and over again that it is important to understand one’s enemies, including their tactics, skills, tools, and motivations.
• The more you know about what your enemy is up to, the better idea you have as to what protection mechanisms you need to put into place to defend yourself.
• Attacks are not only getting more specific, but also increasing in sophistication.
• Malware is still one of the main culprits that costs companies the most amount of money.
• In 2006, Australia's CERT announced that 80 percent of antivirus software products commonly missed new malware attacks because attackers test their malware software against the most popular antivirus software products in the industry to hide from detection.
• A conservative estimate from Gartner pegs the average hourly cost of downtime for computer networks at $42k.
• Today, potentially millions of computers are infected with bots that are controlled by specific hackers.
• A zero-day attack is one for which there is currently no fix available and whoever is running the particular software that contains that exploit vulnerability is exposed with little or no protection.
• WIthin each and every organization, there is the all-too-familiar battle of functionality vs security.
• Security officers are in charge of ensuring the overall security of the environment, which usually means reducing or shutting off many functionalities that users love.
• A vulnerability assessment is usually carried out by a network scanner on steroids.
• Vulnerability assessments are great for identifying the foundational security issues within an environment, but many times, it takes an ethical hacker to really test and qualify the level of risk specific vulnerabilities pose.
• No security professional should ever try to embarrass a customer or make them feel inadequate for their lack of security.
• The goal of a vulnerability test is to provide a listing of all the vulnerabilities within a network. The goal of a penetration test is to show the company how these vulnerabilities can be used against it by attackers.
• Just because a client asks for it, doesn’t mean that it’s legal.
• A computer is just a new tool to carry out old crimes.
• Knowledge and the implementation of knowledge are the keys for any real security to be accomplished.
• In most instances, the toolset used by malicious attackers is the same tools used by security professionals.
• Nothing should be trusted until it is tested.
• Hacking into a system or environment is almost always carried out by exploiting vulnerabilities in software.
• Software, in general, is very complicated, and the more functionality that we try to shove into applications and operating systems, the more complex software will become. THe more complex software gets, the harder it is to predict properly how it will react in all possible scenarios, which makes it harder to secure.
• A common estimate used in the industry is that there are between 5-50 bugs per 1,000 lines of code.
• Every industry in the world is becoming more reliant on software and technology.
• The hacker community’s skill sets are continually increasing.
• Unfortunately, almost all of today’s software products are riddle with flaws.
• CERT/CC is a federally funded research and development operation that focuses on internet security and related issues.
• The Common Vulnerabilities and Exposures (CVE) list is a compilation of publicly known vulnerabilities.
• Effective security cannot be based on obscurity.
• A common but insecure practice for may software vendors is to ship software with backdoors, utilities, and administrative features that help the receiving administrator learn and implement the product.
• Security should be a core discipline when designing the product, during specification and development phases, and during testing phases.
• Highly trained developers create more secure products.
• Social engineering is a way to get someone to do something they wouldn’t normally do for you [...] by creating a false trust relationship with them.
• Emotion is what derails security policy and practices, by leading the human user to make an exception to the rules for what they believe is a good reason.
• If the social engineer looks authoritative and unapproachable, the target usually takes the easy way out by doing what’s asked of them and avoiding a conflict.
• No matter what emotional button the attacker is attempting to push, the premise is always the same: the intended victim will not sense the risk of their action or guess the real intentions of the attacker until it’s too late or, in many cases, not at all.
• Google is probably the most effective way to start finding names, job titles, contact information, and more.
• Finding employees with accounts on popular social media sites is a common practice among social engineers.
• While natural charisma is a prized resource, a practiced phone voice and the ability to discuss convincingly a wide variety of not necessarily technical social topics will get you pretty far down the road.
• A good place to start your reconnaissance after researching the company online is to begin targeting people of interest internally in an attempt to build a picture of who is who and, if possible, develop rapport with potential sources.
• Planning an attack takes time, practice, and, above all, patience.
• Port 443 is a common port to use as it is difficult to proxy and monitor, as the legitimate traffic that would typically flow over it is encrypted.
• By being knowledgeable and conversant in company matters with the information you’ve collected from your social media assets, you can easily build rapport and trust with the employees at the target company online and in person while on site.
• In order to successfully mount a face-to-face SEA, you must not only look the part you’re playing, but also appear as comfortable as you would if you were having a relaxed conversation with a friend. Ideally you want your attitude to put people at ease.
• The most useful metric for determining how calm you are is your heart rate.
• Try this exercise: As you walk in public and encounter people, look them directly in the eye and hold eye contact with them until they break it or you move past them.
• It’s advisable to have  abase script to work from and then deviate as circumstances necessitate.
• A common beginner mistake is to not have something to do with your hands.
• Hardening your environment to withstand SEAs, especially targeted ones, is more a matter of training than a traditional security control.
• An SEA goes right to the most vulnerable point in a company’s defenses: it’s employees.
• The best defense against SEAs is awareness training and simulated targeted attacks.
• Conducting an attack after hours is not recommended. Doing so is extremely dangerous because you might be met by a third party with an armed response or attack dogs.
• You should always have a contact within the target organization who is aware of your activities and available to vouch for you should you be caught.
• It’s a good idea to ask your client in advance to act as if they don’t know you if they encounter you on the premises.
• You have to study any potential target prior to attempting a physical penetration.
• Getting close enough to determine what kind of physical access controls are in place will be helpful in planning your attempt to subvert them.
• The front entrance to any building is usually the most heavily guarded. It’s also the most heavily used, which can be an opportunity.
• Secondary entrances such as doors leading to the smoker’s area and loading docks usually offer good ingress opportunities, as do freight elevators and service entrances.
• When you survey the target site, note how people are entering and exiting the building.
• You should closely examine the front door and lobby; choose someone from your team to walk in and drop off a handful of takeout menus from a nearby restaurant. This will give you some idea of how sophisticated their security controls are and where they’re located.
• If you’ve encountered a professional security guard, he will remember your face, because he’s been trained to do so as part of his job.
• Unless the target organization is large enough that it has its own cafeteria, employees will frequent local businesses for lunch or morning coffee. This is a great opportunity to see what their badges look like and how they wear them.
• Because the smoker’s door is a relatively active area and mostly used for one specific purpose, it represents an excellent opportunity to enter a building unnoticed, or at least unchallenged.
• If you wish to fully understand the lobby security process for a specific building prior to attempting to subvert it, make an appointment with another tenant in the building.
• A mantrap is a two-door entry system. The entrant is allowed through the first door, which then closes and locks. Before the second or inner door unlocks and opens, the entrant must identify and authenticate himself. If he does not, he’s trapped between the two doors and must be released by the security guard.
• Properly implemented and operated, a mantrap cannot be directly subverted except by impersonation.
• When confronted with a mantrap, find a different way in or talk your way past it using the pretense that you are a visitor.
• Every door opens for the fire inspector! Since these positions are often municipal and un-uniformed, they are easily impersonated.
• Before you attempt to defeat a mechanical lock, it’s important to understand how a basic cylinder lock and key work.
• Perhaps the single most effective policy to ensure that an introducer is noticed is one that requires employees to report or inquire about someone they don’t recognize.
• With the exception of the very small company, hired employees are essentially strangers a company pays to perform a task.
• The higher the privilege level of the user, the more trust that is placed in that person and the more risk that is incurred by the company.
• An easy way to see what users are members of the local Administrators group of an individual machine is to use the built-in net command from the command prompt: net localgroup Administrators
• The easiest way to gain access to the Administrator account is to reset its password.
• Offline NT Password is a stripped-down version of Linux with a menu-driven interface. By default, it steps you through the process of removing the Administrator account password.
• Despite widely publicized best practices, in more cases than not the LAN Manager (LM) hash for the Administrator account will still be present on the local machine. This hash can easily be cracked to reveal the local Administrator account password.
• If you are having difficulty uninstalling the AV product, try booting into Safe Mode. This will limit which applications are loaded to a minimum, which in many cases will negate the active protective controls built into AV products allowing you to uninstall them.
• You can use the Sysinternals Process Explorer, procexp, to identify and suspend the process related to the AV product.
• Enumeration should be performed against domain controllers because these servers are responsible for authentication and contain lists of all users in each domain.
• Google Desktop can also help pinpoint obscure file storage directories that may or may not have been noticed any other way during the testing process.
• Because BackTrack [Kali] is a pen-testing distribution, networking services don't’ start by default at boot.
• Metasploit can pair any Windows exploit with any Windows payload.
• The Metasploit Meterpreter is a command interpreter payload that is injected into the memory of the exploited process and provides extensive and extendable features to the attacker. This payload never actually hits the disk on the victim host; everything is injected into process memory with no additional process created.
• A good target is the user’s explorer.exe process. Explorer.exe is the process that manages the desktop and shell, so as long as the user is logged in, explorer.exe should remain alive.
• When pen-testing, your goals will often be to elevate privileges, establish a stronger foothold, and expand across to other machines.
• Administrators tend to reuse the same password on multiple computers, especially when they believe the password to be difficult to guess.
• Scope is probably the most important issue when planning a penetration test.
• Most organizations use a Statement of Work (SOW) when contracting outside work. The format of the SOW is not as important as its content.
• Whenever possible, have the client give you a “get-out-of-jail-free letter”.
• Bad news does not get better with time.
• The Dradis Server is the best way to collect and provide information sharing during a penetration test.
• The strcpy command is probably the most dangerous command used in C.
• The .text section basically corresponds to the .text portion of the binary executable file. It contains the machine instructions to get the task done.
• The .data section is used to store global initialized variables.
• The below stack section (.bss) is used to store global non initialized variables.
• The heap section is used to store dynamically allocated variables and grows from the lower-addressed memory to the higher-addressed memory. The allocation of memory is controlled through the malloc() and free() functions.
• The stack section is used to keep track of function calls (recursively) and grows from the higher-addressed memory to the lower-addressed memory on most systems.
• Local variables exist on the stack section.
• The environment/arguments section is used to store a copy of system-level variables that may be required by the process during runtime.
• The memory space of a process looks like this: text | data | bss | heap -> | unused | <- env="" li="" stack="">
• The term buffer refers to a storage place used to receive and hold data until it can be handled by a process.
• Simply put, strings are just contiguous arrays of character data in memory. The string is referenced in memory by the address of the first character. The string is terminated or ended by a null character.
• Pointers are special pieces of memory that hold the address of other pieces of memory.
• Moving data around inside of memoir is a relatively slow operation. It turns out that instead of moving data, it is much easier to keep track of the location of items in memory through pointers and simply change the pointers.
• To read the value of the memory address pointed to by the pointer, you dereference a pointer with the * symbol.
• Registers are used to store data temporarily.
• The mov command is used to copy data from the source to the destination.
• The add command is used to add the source to the destination and store the result in the destination.
• The sub command is used to subtract the source from the destination and store the result in the destination.
• The push and pop commands are used to push and pop items from the stack.
• The call command is used to call a procedure (not jump to a label). The ret command is used at the end of a procedure to return the flow to the command after the call.
• The lea command is used to load the effective address of the source into the destination.
• The int command is used to throw a system interrupt signal to the processor. The common interrupt you will use is 0x80, which is used to signal a system call to the kernel.
• Hacking tools (any many other applications) use Python because it is a breeze to learn and use, is quite powerful, and has a clear syntax that makes it easy to read.
• Dictionaries are similar to lists except that objects stored in a dictionary are referenced by a key, not by the index of the object.
• Dictionaries are a great way to store any values that you can associate with a key where the key is a more useful way to fetch the value than a list’s index.
• In Python, white space matters, and indentation is used to mark code blocks.
• The stack is one of the most interesting capabilities of an operating system.
• In memory, each process maintains its own stack within the stack segment of memory.
• Buffers themselves have no mechanism to keep you from putting too much data in the reserved space.
• When dealing with buffer overflows, there are basically three things that can happen. The first is denial of service. The second thing that can happen when a buffer overflow occurs is that eip can be controlled to execute malicious code at the user level of access. This happens when the vulnerabile program is running at the user level of privilege. The third and absolute worst thing that can happen when a buffer overflow occurs is that the eip can be controlled to execute malicious code at the system or root level.
• Local exploits are easier to perform than remote exploits because you have access to the system memory space and can debug your exploit more easily.
• The basic concept of buffer overflow exploits is to overflow a vulnerability buffer and change eip for malicious purposes.
• In assembly code, the NOP command simply means to do nothing but move to the next command.
• If eip is pointed to a NOP sled, the processor will ride the sled right into the next component.
• On x86 systems, the 0x90 opcode represents NOP.
• The most important element of the exploit is the return address, which must be aligned perfectly and repeated until it overflows the saved eip value on the stack.
• As a rule of thumb, it is a good idea to fill half of the attack buffer with NOPs.
• What happens when the vulnerable buffer is too small? The answer lies in the use of environment variables. You would store your shellcode in an environment variable or somewhere else in memory, then point the return address to that environment variable.
• Unlike buffer overflows, format string exploits are relatively easy to spot in source code and binary analysis.
• In C/C++, the destructor (DTOR) section provides a way to ensure that some process is executed upon program exit.
• It turns out that if we overwrite either an existing function pointer in the DTOR section or the ending marker with our target return address, the program will happily jump to that location and execute.
• There are many other useful locations to overwrite; for example: global offset table, global function pointers, atexit handlers, stack values, program-specific authentication variables.
• Libsafe is a dynamic library that allows for the safer implementation of [many] dangerous functions.
• Libsafe overwrites dangerous libc functions, replacing the bounds and input scrubbing implementations, thereby eliminating most stack-based attacks.
• StackShield is a replacement to the gcc compiler that catches unsafe operations at compile time.
• In addition, when a function is called, StackShield copies the saved return address to a safe location and restores the return address upon returning from the function.
• SSP [Stack Smashing Protector] has been incorporated in GCC (starting in version 4.1) and is on by default. It may be disabled with the -fno-stack-protector flag.
• GCC has implemented a non-executable stack using the GNU_STACK ELF marking. This feature is on by default (starting in version 4.1) and may be disabled with the -z execstack flag.
• Early on, developers realized that program stacks and heaps should not be executable and that user code should not be writeable once it is placed in memory.
• The intent of ASLR is to randomize the following memory objects: executable image, brk()-managed heap, library images, mmap()-managed heap, user space stack, kernel space stack.
• Systems that implement ASLR provide a high level of protection from “return to libc” exploits by randomizing the way the function pointers of libc are called. This is done through the randomization of the mmap() command and makes finding the pointer to system() and other functions nearly impossible.
• “Return to libc” is a technique that was developed to get around non-executable stack memory protection schemes such as PaX and ExecShield. Basically, the technique uses the controlled eip to return execution into existing glibc functions instead of shellcode.
• It turns out that functions like system() and exit() are automatically linked into binaries by the gcc compiler.
• Reliable shellcode is at the heart of virtually every exploit that results in “arbitrary code execution”, a phrase used to indicate that a malicious user can cause a vulnerable program to execute instructions provided by the user rather than the program.
• User-space is that portion of a computer’s memory space dedicated to running programs and storing data that has no need to deal with lower-level system issues. That lower-level behavior is provided by the computer’s operating system, much of which runs in what has come to be called kernel space, since it contains the core, or kernel, of the operating system code and data.
• Programs that run in user space and require the services of the operating system must follow a prescribed method of interacting with the operating system, which differs from one operating system to another.
• In generic terms, we say that user programs must perform “system calls” to request that the operating system perform some operation on their behalf.
• Virtually all significant capabilities required by shellcode are controlled by the operating system, as such, it is important for shellcode authors to understand how to access these services on the platforms for which they are authoring shellcode.
• Windows shellcode must go through a discovery process to locate each function that it needs to call before it can call those functions.
• Regardless of the operating system that we are targeting, processes are provided three open files when they start. These files are typically referred to as the standard input (stdin), standard output (stdout), and standard error (stderr) files.
• As an attacker, you must ensure that before you create a shell process, you have properly set up your input/output file descriptor(s) to become the stdin, stdout, and stderr that will be utilized by the command shell once it is launched.
• In many cases, firewalls are less restrictive regarding outgoing traffic.
• Reverse shellcode, also known as “callback shellcode”, exploits this fact by reversing the direction in which the second connection is made.
• The most common technique used in shellcode for locating the proper socket descriptor is to enumerate all of the possible file descriptors in the vulnerable application, and to query each descriptor to see if it is remotely connected to our computer.
• Multistage payloads generally consist of two or more stages of shellcode, with the sole purpose of the first (and possible later) stage being to read more shellcode and then pass control to the newly read-in second stage, which, we hope, contains sufficient functionality to carry out the majority of the work.
• Launching new processes, creating new network connections, and creating new files are all actions that are easily detected by security-conscious system administrators.
• A system call (or syscall) proxy is a small piece of shellcode that enables remote access to a target’s core operating system functionality without the need to start a new process like a command interpreter such as /bin/sh. The proxy code executes in a loop that accepts one request at a time from the attacker, executes that request on the target computer, and returns the results of the request to the attacker.
• Conceptually, it is as if the hostile program were actually running on the target computer, yet no file has been uploaded to the target, and no new process has been created on the target, as the system call proxy payload can continue to run in the context of the exploited process.
• Process injection shellcode allows the loading of entire libraries of code running under a seperate thread of execution within the context of an existing process on the target computer.
• Whenever we attempt to exploit a vulnerable application, it is important that we understand any restrictions that we must adhere to when it comes to the structure of our input data.
• Determining exactly which characters must be avoided typically is accomplished through a combined process of reverse-engineering an application and observing the behavior of the application in a debugging environment.
• The purpose of a shellcode encoder is to transform the bytes of a shellcode payload into a new set of bytes that adheres to any restrictions imposed by our target application.
• A very important thing to understand about shellcode is that, like any other code, it requires storage space while executing.
• Stability becomes a huge concern when developing kernel-level exploits.
• Any shellcode you use needs to take into account the effect your exploit will have on the threat that you exploited.
• Proper cleanup is a very important piece of any kernel exploit.
• The purpose of the operating system is to serve as a bridge between the user (process) and the hardware.
• If there are any null characters, the shellcode will fail when we place it into a string for injection during an exploit.
• Sockets are defined as the binding of a port and an IP address to a process.
• The XOR function is interesting because it is reversible, meaning if you XOR a number with another number twice, you get the original number back as a result.
• The reversible characteristics of the XOR function make it a great candidate for encoding and basic encryption.
• When shellcode is encoded, a decoder needs to be placed on the front of the shellcode.
• One of the most common GETPC techniques is the JMP/CALL technique. We start with a JMP instruction forward to a CALL instruction, which is located just before the start of the encoded shellcode. The CALL instruction will push the address of the next address of the next address onto the stack and jump back to the next instruction. At that point, we can pop the location of the encoded shellcode off the stack and store it in a register for use when decoding.
• Another popular GETPC technique is to use the FNSTENV assembly instruction as described by noir. The FNSTENV instruction writes a 32-byte floating-point unit (FPU) environment record to the memory address specified by the operand.
• The msfpayload command is supplied with Metasploit and automates the generation of shellcode.
• The /GS switch enables Microsoft’s implementation of stack canary protection, which is quite effective in stopping buffer overflow attacks.
• Recall that the exploit development process is as follows: control eip, determine the offsets, determine the attack vector, build the exploit sandwich, test the exploit, debug the exploit if needed.
• When [Windows] programs crash, the operating system provides a mechanism to try to recover operations, called structured exception handling (SEH). This is often implemented in the source code with try/catch or try/exception blocks.
• Windows keeps track of SEH records by using a special structure.
• The /GS compiler option is the Microsoft implementation of a stack canary concept, whereby a secret value is placed on the stack above the saved ebp and saved RETURN address. Then, upon return of the function, the stack canary value is checked to see if it has been changed.
• The purpose of the SafeSEH protection is to prevent the overwrite and use of SEH structures stored on the stack.
• In the past, a traditional heap exploit would overwrite the heap chunk headers and attempt to create a fake chunk that would be used during the memory-free routine to write an arbitrary 4 bytes at any memory address.
• Data Execution Prevention (DEP) is meant to prevent the execution of code placed in the heap, stack, or data sections of memory.
• The DEP settings for an application are stored in the Flags bitfield of the KPROCESS structure, in the kernel.
• The purpose of address space layout randomization (ASLR) is to introduce randomness (entropy) into the memory addresses used by a process. This makes attacking much more difficult, as memory addresses keep changing.
• The /GS protection mechanism uses several weak entropy sources that may be calculated by an attacker and used to predict (or guess) the cookie value.
• It turns out that the /GS protection does not protect the SEH structures placed on the stack.
• The easiest way to bypass ASLR is to return into modules that are not linked with ASLR protection.
• PDF is now the most commonly attacked content file type.
• Most PDF-based vulnerabilities in the wild exploit coding errors made by Adobe Reader’s JavaScript engine.
• You should always change the file extension of potentially malicious samples. When handling malicious EXE samples, changing the file extension prevents accidental execution.
• Remember that real-world exploits are binary, obfuscated, compressed, and jumbled up.
• The vast majority of content-type attacks attempt to exploit already-patched vulnerabilities.
• Simply applying all security updates blocks most content-type attacks [...].
• Data Execution Prevention (DEP) is an effective mitigation against many real-world exploits.
• Web application injection vulnerabilities result from poor input validation.
• Applications are vulnerable to cross-site scripting (XSS) when they permit untrusted, attacker-provided data to be actively displayed or rendered on a web page without being escaped or encoded.
• Any web application that accepts user input as the basis for taking action or performing a database query may be vulnerable to a SQL injection.
• Strict input validation prevents injection vulnerabilities.
• A common character used as a simple check for SQL injection is a single quote.
• Databases store data in a structured manner that allows easy retrieval and cross-referencing. Organizing the data in a “relational” manner makes it easier to query and retrieve any data in the database. Relational databases store data in tables organized by rows and columns. Entries in different tables can cross-reference each other via a unique identifier for each row.
• XSS vulnerabilities primarily impact the users of the web application, not the web application itself.
• VoIP, or Voice over Internet Protocol, is a type of transmissions medium that is responsible for the devliery of real-time voice and data communication.
• Enumeration is the process of gathering informain about a target system or network entity for reconnaissance and potential exploitation.
• One type of attack that can consuem resources and cause outages is the SIP invite flood.
• In our opinion, VoIPER is one of the most comprehensive open source VoIP test tools ebcause it is based on the concept of fuzzing, a technique that is used to inject irregular message content and data inputs in an effort to test the robustness of a system.
• Test your system thouroughly via penetration testing and implement a strategy of defense in depth that encompasses the entire system.
• Almost everything is connected to the vast realm of the internet, and SCADA devices are no exception.
• SCADA stands for supervisory control and data acquisition.
• SCADA networks control and monitor the critical utility and process contro infrastructures for manufacturing, production, and power generation for utility companies, including electricity, natural gas, oil, water, sewage, and railroads.
• Since the inception of telemetry, SCADA networks have become popular to control electrical and other infrastructure systems.
• SCADA uses several protocols. The most common protocols are: object linking and embedding for process control (OPC), inter-control center protocol (ICCP), modbus, and distributed network protocl version 3 (DNP3).
• Modbus is a protocol specification designed for building automation equpiment used to interface with various devices over RS485 serial and TCP/IP interfaces.
• SCADA devices are prone to the same common vulnerabiltites--such as enumeration, password cracking, network evasdropping, and denial of service--that are found in any other types of network devices.
• Fuzzing provides an intelligent approach to injecting irregular message content and data inputs in an effort to qualify the robustness of a system.
• The key here [to fuzzing] is to try every kind of malformed message trigger that we can think of, to see if we can cause some sort of abnormality or disruption.
• Just about any protocl has the potential to be reverse engineered and used for malicious intent.
• Many programming lanauges allow the programmer to ignore the values returned by functions. This is a dangerous practice because function return values are often used to indicate error conditions.
• Preconditions are a set of one or more conditions that must be true upon entry into a particular portoin of a program.
• Postconditions are a set of conditions tht most hold upon exit from a particular section of a program.
• The strcpy() function is dangrous because it copies data into a destination buffer without any regard for the size of the buffer and therefore may overflowthe buffer.
• It is important to make sure that proper validation of input data is taking place.
• For all but the most trivial of programs, it is virtualy impossible to formally prove that a program is secure.
• The level of effort requied to determine whether a potential trouble spot is vulnerable is generally much higher than the level of effort the white hat will expend fixing that same trouble spot.
• Since most vulnearbilites are exploited when programs fail to properly handle user input, it is important to understand first how data is passed to an application, and second what happens with that data.
• To locate vulnerabilities, you need to determine which types of input, if any, result in user-supplied data being manipulated in an insecure fashion.
• Proficiency at reverse engineering binaries rquires patience, practice, and a good collection of reference material.
• The purpose of a disassembler is to attempt to geenrate assembly lanauge from a compiled binary, while the purpose of a decompiler is to attempt to generate source code from a compiled binary.
• Decompilation is perhaps the holy grail of binary auditing.
• Two common executable file formats are the Portable Executable (PE) file format used for Microsoft Windows executables, and the Executable and Linking Format (ELF) used by Linux and other Unix variants.
• A particulalry impressive feature of IDA Pro is its ability to track program stack usage within each recognized function.
• Dissassemblies are made more readable when strucutre names are used rather than register plus offset syntax.
• The process of manually searching for vulnerabilties using IDA Pro is similar in many respects to searching for vulnerabilites in source code. A good start is to locate the places in which the program accepts user-provided input, and then attempt to undetstand how that input is used.
• BinNavi from Zynamics is a tool that provides for graph-based analysis and debugging of binaries.
• BinDiff, as its name implies, displays the differences between two versions of the same binary.
• BinDiff combines disassembly with graph comparison algorithms to compare the control flow graphs of successive versions of functions and highlights the newly introduced code in a display format similar to that of BinNavi.
• The process of “stripping” a binary invovles removing all symbol information that is no longer required once the binary has been built.
• A significant amount of initialization must take place before control can be transfered to main.
• Programs that use shared libraries are said to be dynamicaly linked, while programs that use static libraries are said to be statically linked.
• Dynamic linking results in smaller executables and easier upgrading of library components at the expense of some extra overhead when launching the binary, and the chance that the binary will not run if any required libraries are missing.
• Static linking results in much larger binaries becauyse library code is merged with program code to create a single executable file that has no external dependies, making the binary easier to distribute.
• The last thing you want to do is spend your time reversing library code that is generally accepted to be fairly secure.
• One consequence of compilation being a lossy operation is that we lose acces to data declarations and structure definitions, which makes it far more difficult to understand the memory layout in dissasmbled code.
• There are two important steps in determining the layout of data structures in compiled code. The first step is to determine the size of the data structure. The second step is to determine how the structure is subdivided into fields and what type is associated with each field.
• There are two methods for determing the size fo a structure. The first and easietst method is to find locations at which a strucutre is dynamically allocated using malloc or new. The seoncd method of determining the size of a structure is to observe the offesets used in every referenceto the structure and to compute the maximum size required to house that data that is referenced.
• To understand the layout of the bytes within a strucutre, we must determine the types of data that are used at each observable offset within the structure.
• C++ is a somewhat more complex language than C, offering member functions and polymorphism, among other things. These two features require implemention dtails that make compiled C++ code look rather different from compiled C code when they are used. First, all nonstatic member functions require a ‘this’ pointer; and second, polymorphism is impleented through the use fo vtables.
• Virtual tables (vtables) are the mechanis underlying virtual functions and polymorphism in C++.
• A vtable contains an entry for each virtual function in a class, and the compiler files each entry with a pointer to the virtual function’s implementation. Subclasses that override any virtual functions each receive their own vtable.
• By examining similaraties among vtables, it is possible to understand inheretance relationships among classes in a C++ program.
• A security conscious end user should always assuem that there are problems that have avoided detection all the way through the testing phase.
• A tremoudous number of software bugs are found simply because a user provided unexpected input to a program.
• Fuzzing is one of the main tecniques used in black/gray box testing. To fuzz effectively, two types of tools are required, instrumentation tools and fuzzing tools. Instrumentation tools are used to pinpoint problem ares in programs either at runtime or during post-crash analysis. Fuzzing tools are used to automatically generate large numbers of interesting input cases and feed them to programs.
• The challenge to the tester is to ensure that all code paths behave predicteibley under all input cases. To do this, test cases must be developed that force the program to execute all possible instructions within the program.
• Threaded programs and programs that fork can be difficult for debuggers to follow.
• Following a fork operation, you must decide whether to follow and debug the child process or to stick with and continue debugging the parent process.
• A core dump is simply a snapshot of a process’s state, including memory contents and CPU register values, at the time an exception occurs in a process.
• To monitor kernel-level software such as device drrivers, kernel-level debuggers are required.
• Code coverage tools give developers an idea of what portions of their programs are actually getting executed. Such tools are excelletn aids for test case development.
• Some of the most usefyul tools for black box testing are those that monitor the way that a program uses memory at runtime.
• A double free condition occurs wen the free function is called a second time for a poitner that has already been freed. The second call to free corrupts heap management information that can result in an exploitable condition.
• Should you choose to fuzz a program, Valgrind can be a critical peice of instrumentation that can help to quicly isolate memory problems, in particular heap-based buffer overflows, which manifest themselves as invalid reads and writs in Valgrind.
• The real challenge of fuzzer development is building them in such a way that they generate intersting input in an intelligenct, efficient manner.
• To reach the many possible code paths for a given problem, a fuzzer usually needs to be somewhat “protocol aware”.
• Fuzzing should generally be performed with some form of instrumentation in place.
• The goal of fuzzing is to induce an observable error condition in a program.
• Without resorting to reverse enginering a program binary, one of the few ways you can hope to learn about an unknown protocol is by observing communicaionts to and from the program.
• Vulnerable SUID root binaries can provide an easy means for local privelage escalation attacks.
• Client-side vulnerabilites exploited for code execution result in attack code exeecuting at the same pirvlege level as the client-side application executes normally.
• The Windows heap manager itself by default does not zero out memory between uses. It could, but that would incur a performance hit.
• It’s important to point out that most real-world compromises are not due to zero-day attacks. Most compromises are the result of unpatched workstations.
• Access control is about the science of protecting things.
• Not all vulnerabilities in access control are this easy to exploit, but once you undrestand the concepts, you’ll quickly understand the path to privelege escalation, even if you don’t yet know how to take control of execution via a buffer overrun.
• Windows access tokens work in a similar manner as an employee badge. The access token is a container of all a user’s security information and is checked when that user requests access to a secured resource.
• Every process gets its own token describing the user context under which the process is running.
• Having per-process tokens is a powerful feature that enables scenarioes that would otherwise be impossible.
• The two easiest ways to dump the access token of a process or thread are Process Explorer and the !token debugger command.
• Any program that autostarts is interesting to us because it will likely be autostarted in the security context of a highly privileged account.
• Shared memory sections are blocks of memeory set aside to be shared between two applications. This is an especially handy way to share data between a kernel-mode process and a user-mode process.
• The problem with basic fuzzing is that you often only scratch the surface of a server’s interfaces and raregly get deep inside the server to find bugs.
• Crashability and exploitability are vastly different things.
• During [crash] analysis, the instruction pointer is often a good place to start looking for problems.
• For the purposes of debugging, remeber that eip is always pointing at the next instruction to be executed.
• If you haven’t managed to take control of eip, the next step is to determine what damage you can do using other available registers.
• Common overwrite locations include saved return addresses, jump table pointers, import table pointers, and funciton pointers.
• Preconditions are those conditions that must be satisfied to properly inject your shellcode into a vulnerable application.
• Postconditions are the things that must take place to trigger execution of your code once it is in place.
• Locating a ‘jmp esp’ or other jump to register is your best defense against a shifting stack, including ASLR-associated shifts.
• The basic idea behind a return-to-libc exploit is to overwrite a saved return address on the stack with the address of an interesting library function.
• If you can return to a function such as system(), you can execute virtually any program available on the victim system.
• It is always wisest to assume that someone will discover or learn of the same vulnerability we are investigating before the vulnerability is patched.
• Port knocking is a defensive technique that can be used with any network service but is most effective when a service is intended to be accessed by a limited number of users.
• Port knocking is probably best described as a network cipher lock.
• The basic idea behind port knocking is that the port on which a network service listens remains closed until a user steps through a required knock sequence. A knock sequence is simply a list of ports that a user attempts to connect to before being granted permission to connect to the desired service.
• The only sure way to secure a vulnerable application is to shut it down or patch it.
• One tool for generating and applying binary patches is named Xdelta. Xdelta combines the functionality of diff and patch into a single tool capable of being used on binary files.
• One of the toughest problems to overcome when patching a binary is finding space to insert new code.
• An absolute address is an unambiguous location assigned to an instruction or to data.
• A relative offset describes a location as the distance from some reference location.
• In the case of a heap overflow, the attacker’s goal is to overwrite heap control structures with specially chosen values that will cause the heap management routines to write a value of the attacker’s choosing into a location of the attacker’s choosing.
• Malware can be defined as any unintended and unsolicited installation of software on a system without the user knowing or wanting it.
• One of the most important aspects of a piece of malware is its persistence after reboots and its longevity.
• Packers are used to “pack” or compress the Windows PE file format.
• Any time you introduce another system onto the network, you impose a new risk on the network.
• The first thing you need to do with a foreign binary is determine what type of file it is.
• The PEiD tool is very useful in telling you if the file is a Windows binary and if the file is compressed, encrypted, or otherwise modified.
• The ultimate form of static analysis is reverse engineering.
• The FileMon program is very useful in finding changes to the file system.
• The Process Explorer tool is very useful in examining running processes.
• Recovery of any decryption keys is an essential step for reverse-engineering any encrypted malware.
• One of the most prevalent features of modern malware is obfuscation.
• Obfuscation is the process of modifying something so as to hide its true purpose.
• Debuggers rely on the ability to process specific CPU exceptions.
• The first actions that most malware takes generally center on survival.
• Automated malware analysis is a difficult problem.