Organisations have been collecting data for decades but only relatively recently has the volume of data increased beyond that of manual analysis. The cost of analysis can often be non-linear.
Data available includes not only local data but data collected from other organisations, both friendly and hostile. This data can be used for comparison or augmentation of the datasets already collected.
The Cost of Data Storage
Reducing rapidly and has obeyed Moore’s Law for 30 years now.
Means that the storage of data is not economically prohibitive. Organisations can easily retain historical data.
Storage Space affordability has been doubling every 18 months
Storage Space affordability has been doubling every 18 months
New devices - eg. flash memory
Access speeds have been increasing by ~10% pa. (Note, as an aside, that seek time has been increasing less quickly than transfer rates).
Management is increasingly realising that knowledge of an environment gives strategic advantage.
Staff Shortages and Staff Turnover
Both of these affect the knowledge-base of an organisation. Staff turnover is increasing.
Complexity of Data
In the past data were relatively simple. The availability of historical and geo-referenced data, for example, has meant that new analytical methods must be employed or some knowledge may be missed.
Greater volumes of more complex data.
Greater volumes of more complex data.
The devices to store the data on at affordable prices.
A recognition that there maybe something to be learnt from that data.
A knowledge-drain from the organisation.
Processing power to burn, particularly out-of-hours.
Some intelligence is passive (ie. understanding of a situation), other intelligence is active (the ability to act in a given situation).
Some intelligence is passive (ie. understanding of a situation), other intelligence is active (the ability to act in a given situation).
The distinction is important when we determine what to consider “interesting”.
A thing is identical to itself; (the Law of Identity),
A thing cannot be and at the same time not be (the Law of Contradiction),
A thing either has property X of does not have property X (Law of the Excluded Middle).
David Hume suggested that every claim or proposition can be categorised into one of three categories (known as Hume's Fork):
True or false by definition;
True or false depending on experience;
Nonsense.
Generally speaking, computer systems used to largely make decisions using deductive logic.
Generally speaking, computer systems used to largely make decisions using deductive logic.
A set of axioms (which are by definition correct) are applied to known facts to produce new facts.
Data recorded in a database is often considered deductively correct.
Note that relational database data is really just a set of deductive logical facts:
HasEmployeeNo (Bob, 98734J)
AssignedtoDept (98734J, Sales) … etc.
Humans, on the other hand, generally make decisions based on either inductive logic or irrationality.
Humans, on the other hand, generally make decisions based on either inductive logic or irrationality.
Irrational behaviour will not be covered further except to note that humans do not always act rationally - as much as we would like them to!
For induction, we derive a set of heuristics from our experiences and take decisions based on these.
I arrive at the bus stop early as the bus often runs early.
I take notes during the lecture despite the notes being available as it helps me remember things better.
I will do my assignment before the break as I will forget it otherwise.
Induction (including scientific fact)works by evidence. In all the examples above there are exceptions but a heuristic (a rule of thumb) can sometimes be derived.
