Another decision point is how long to store the logs. This needs to be carefully considered, especially as there are legal requirements you may be subject to or industry-specific rules that apply to you.
For example, PCI-DSS requirements ask you to store the PCI-scoped logs for a year. Country-specific rules may require you to delete logs after a certain period of time so as to respect privacy.
The typical tradeoffs:
Keep the data for a long time:
* Higher likelihood that you'll have the log required to solve the crime.
* Higher storage requirement and performance impact to manipulate lots of data.
Keep the data for a short time:
* Maybe you'll have the log if you need it, but maybe not.
* Easy on your storage, and better performance.
The right approach: As I've mentioned before, the right approach would be to apply a risk-management method. You first need to identify the legal and industry constraints that apply, which will give you a minimum/maximum range, then you need to understand how far back in time you want go for your forensics. Again, there is no one-size-fits-all solution for this.
As a rule of thumb, keep the logs as long as possible while respecting legal and industry requirements and respecting privacy issues.
Storing logs securely
You need to trust the logs that you are using for forensics and in case of prosecution you also need to prove to a court of law that the logs are genuine and that nobody has tampered with them. How?
There are actually two types of integrity you need to prove.
* Integrity of each raw log. This will prove no log has been tampered with or manipulated
.*Integrity of the log sequence. This will prove no log has been added and no log has been deleted.
This is not easy to do. For example, signing each log will guarantee log integrity but not the integrity of the log sequence.
At this point you can rule out most homegrown log management solutions, and you can throw away most open source solutions; in fact, there are not many solutions that are capable of providing both of these.
One way of proving both log integrity as well as log sequence integrity is to store raw logs in flat files that are digitally signed or at least hashed using a strong hashing mechanism. Pros and cons are as follows.
Store raw logs as is:
* Logs are available in original format for most flexibility in subsequent use.
* Difficult/impossible to guarantee their integrity.
* Storage requirement fits complexity of logs.
Store normalized logs in database:
* Logs are available in "intelligent" form for easy reporting.
* Difficult/impossible to guarantee their integrity.
