An audit of information security at the Department of Justice says that though the agency got an A-plus rating under federal standards, those measure only processes on paper and that, in reality, no one knows how secure computers in the department -- and, by extension, the rest of the federal government -- are.

The audit, by Justice Department Inspector General Glenn A. Fine, also noted that the department "lacks effective methodologies … for maintaining an inventory of devices connected to the department's various (information technology) networks."

The Federal Information Security Management Act of 2002 says all federal departments and agencies must conduct yearly assessments to measure their compliance with information security standards in the act.

In May the Justice Department's compliance was rated A-plus by the U.S. House Committee on Oversight and Government Reform.

But FISMA mandates, as the inspector general's report noted, are primarily concerned with ensuring that all agencies "have policies and procedures to enhance the security of information in their IT systems."

The Justice Department's A-plus grade, therefore, "did not assess whether the Department has actually implemented these processes, nor did it assess the actual security of the Department's IT systems."

"Unfortunately, FISMA has become a compliance exercise," said Shannon Kellogg, director of information security policy for EMC Corp. As a result, he said, "even if an agency receives a good grade, it does not mean that that agency has significantly reduced risks to information security or reduced the number of serious cyber incidents."

Moreover, as the inspector general's report pointed out, although the department had a FISMA-mandated process for conducting regular assessments of vulnerabilities, it "had not fully implemented the policies and procedures intended to remediate identified vulnerabilities."

"These vulnerabilities increase the risk of unauthorized users gaining access to department IT systems and potentially compromising sensitive department information," said Fine in a statement, adding that the department "has struggled to mitigate these vulnerabilities after they are identified."

Details of most of the vulnerabilities were redacted from the report, but auditors did say in a footnote that anti-virus software on department computers was often out of date.

The department, in its response to the report, published Friday, concurred with the findings and recommendations. It stated that, since the audit was conducted, it had already fulfilled one of the recommendations -- for real-time monitoring of department computer networks -- by establishing a security operations center. It also said the complete inventory of devices connected to the department's networks was under way and would be completed next month.

But the report nevertheless raises the question of how to measure IT security on U.S. government systems -- an issue that has dogged officials for years and continues to be a concern, given the thousands of attacks on such networks every year by hackers, spies and potential enemies.

FISMA long has been derided by critics as promoting a "tick the box" approach to IT security.

The Justice Department has had "terribly damaging cyberattacks that were made possible because the agency had spent its budget on FISMA reporting instead of on critical security improvements," Alan Paller, director of research at the SANS Institute, a non-profit cybersecurity research group, told UPI.

"It measures compliance, it measures process, it doesn't measure outcomes," added one IT security industry executive who asked not to be quoted by name. "That is the approach bureaucrats love."

But this process-oriented approach to federal IT security recently has been challenged, in part by a new initiative being promoted by former U.S. Air Force Chief Information Officer John M. Gilligan. The Consensus Audit Guidelines, as they are known, promote an approach that focuses on fixing vulnerabilities in federal networks that hackers most frequently exploit.

"Let's figure out what are the vulnerabilities being exploited and fix those first," Gilligan told a recent IT security conference in Washington, "There should be a focus in the investment on what delivers the greatest payout."

The guidelines promote a "defense that is informed by the offense," added Paller. "You need to address the known bads first."

Paller called the guidelines "a game changer." They will be submitted for public review next year and then be taken up by the White House Office of Management and Budget.

Even FISMA's defenders acknowledge that it needs overhauling, and the Senate Homeland Security and Governmental Affairs Committee approved an updated version of the law in October, which staffers say the committee will be pushing next year.

"The law is now six years old, the threats have evolved, and FISMA needs an overhaul," Kellogg told UPI.

"The incoming Congress should make FISMA reform a priority. We can't afford to address today's quickly changing threats with a legal framework that is six years old. ... There is pretty broad consensus on that point."

The new law seeks to refocus information security efforts on performance and risk management; gives more power to department chief information security officers; and mandates independent annual audits of compliance, rather than the current departmental evaluations.

Gilligan says those changes would dovetail with the Consensus Audit Guidelines.