SMS Audit Key Pointers

Audit pointers for the safety management system (SMS) are guidelines and considerations that auditors use to conduct a thorough and effective audit of regulatory compliance statements and safety in operations. The specific pointers may vary depending on the nature and scope of the audit, but here are some general audit pointers that are applicable in various audit scenarios.

Understand the Business, Safety Management System, and Aviation Industry: An auditor begin their audit by gaining a deep understanding of the audited entity's business operations, SMS and industry as airline or airport. This knowledge is crucial to identifying key risks and areas to focus on during the audit. There are qualifications requirements for an auditor to do financial and tax audits, and the requirement is more than the ability to use a pen and pencil and being able to read and write. However, in the aviation industry, both airports and airlines, these are the only requirements. There are no requirements for an SMS auditor to be approved by the regulator or have superior knowledge and experience of the SMS regulations, standards, and compliance requirements. SMS audits are to be taken with a grain of salt. SMS audits is about processes conformant to regulatory requirements and is not about opinions of operations future impact. 

 Becoming a financial auditor typically requires a combination of education, professional certification, and relevant skills and experience. An SMS auditor is required to be accepted by the accountable executive as a third-party auditor, or an inhouse auditor. Auditing a safety management system is a new type of audit and different than financial or technical audits. An SMS audit is of human factors, organizational factors, supervision factors and environmental factors in addition to regulatory compliance. The audit of an SMS is a performance audit and an audit of its systems and processes, and how processes conform to regulatory requirements, standards, and the SMS policy. The regulatory portion of an audit is an audit without consideration of processes if an SMS enterprise is in compliance with regulations and standards. 

Requirements for financial auditors may be used as guidance for SMS auditors of airports and airlines. 

Bachelor's Degree: Most financial auditors have at least a bachelor's degree in accounting, finance, or a related field. Some auditors may pursue degrees in business administration or economics as well. Following this principle is an asset for SMS auditors. 

CPA (Certified Public Accountant): Many auditors pursue the CPA credential, which is highly respected in the field of accounting and auditing. Becoming a CPA typically requires passing the CPA exam and meeting specific education and experience requirements, which vary by state or jurisdiction. There are third-party certification credentials available for SMS auditors which may be helpful tools. Experience as SMS manager and accountable executive are useful prerequisites for SMS auditors.   

Analytical Skills: Auditors need strong analytical skills to review financial statements, identify discrepancies, and assess the accuracy of financial records. This also true for SMS auditors. 

Attention to Detail: Auditors must have a keen eye for detail to spot errors or irregularities in SMS performance data.

Communication Skills: Effective communication is essential for auditors to explain their findings to clients or management and to document their work in reports.

Knowledge of Accounting Standards: Auditors must be familiar with accounting principles, auditing standards, and financial reporting regulations (e.g., Generally Accepted Accounting Principles or International Financial Reporting Standards). This principle is crucial for SMS auditor, that they have in-depth knowledge of airport and airline regulations and standards, and how processes helps SMS enterprises to achieve regulatory compliance and safety in operations. 

Computer Skills: Proficiency in using auditing software and spreadsheet programs like is important for data analysis and reporting. SPCforexcel.com is an invaluable tool for SMS auditors. When audits are based on other than data and statistical principles special cause variations remain hidden and incorrect corrective action plans are implemented. 

Entry-Level Positions: Many auditors start their careers in entry-level positions, such as staff accountant or junior auditor, to gain practical experience in auditing. A competent SMS auditor must have the following practical experience:

Airside maintainer: principles and systems – airport standards, technical and processes – build an airport, and airside applications – audit an airport.

Airport manager: manual management, daily quality control and project planning.

Accountable executive: an accountable executive is responsible for operations or activities authorized under the certificate and accountable on behalf of the certificate holder for meeting the requirements of the Canadian Aviation Regulations.

At the time of writing NLC at CYDQ offers these courses. 

Progression: As auditors gain experience, they can move on to more senior roles, such as senior auditor, audit manager, or even internal auditor positions within organizations.

Staying up to date with changes in auditing standards and regulations is crucial. Many auditors participate in continuing education and professional development programs to maintain their skills and knowledge. This is crucial for an SMS auditor and requires individuals to monitor daily changes affecting their audit processes.

Auditors are expected to adhere to high ethical standards to maintain the integrity and credibility of the audit process. Independence and objectivity are especially critical. For smaller SMS operators this becomes an issue of personal accountability, when internal workers, or the SMS manager is performing the audit. The regulatory requirement is that SMS audits shall be fulfilled by persons who are not responsible for carrying out those tasks or activities unless the size, nature and complexity of the operations justify the fulfilling of those duties by the person responsible for carrying out those tasks or activities, and a risk analysis, that the fulfilling of those duties by the person responsible for carrying out those tasks or activities will not result in an unacceptable risk to aviation safety. 

Auditing an SMS enterprise is a highly specialized field. The audit of an SMS enterprise is not just to audit the outcome, but to audit the processes that produced the outcome. Just as a financial audit does not accept business expenses at face value but audit the processes generating a profit or loss. An SMS auditor must audit the data and processes applied to justify their result. 

Audit Planning: Develop a comprehensive audit plan that outlines the scope, objectives, and audit procedures. Consider the materiality threshold and risk assessment to determine the level of audit effort needed for different areas. The regulatory requirements are for SMS enterprises to perform an audit of the entire quality assurance program carried out every three years, calculated from the initial audit. The quality assurance program is a quality assurance audit of the entire airport certificate.

Risk Assessment: Identify and assess operational risks that could impact the accuracy of SMS enterprises statements. Focus risk assessment on safety critical areas and safety critical functions and allocate audit resources accordingly. This does not impact the audit itself, since the audit is of the entire certificate, but is to prioritize risk assessments in the audit report. 

Internal Controls: Evaluate the effectiveness of the organization's internal controls, including the design and implementation of controls. Test key controls relevant to the audit. SMS performance assessment is a regulatory requirement for both airports and airlines. In addition, they are required to monitor the concerns of the civil aviation industry in respect of safety and their perceived effect on their operating certificate. Unknown bad news or reputation is a failed audit item. 

Sampling: The only acceptable method to sample is to use random sampling and statistical sampling techniques to select samples for testing. Sampling based on gut-feelings corrupts the audit process. Ensure the sample is representative of the population being tested. When applying statistical principles and statistical process control (SPC), any out-of-control tests requires the SMS enterprise to identify the special cause variation that caused the out-of-control process. 

Document Everything: Maintain detailed audit documentation, including the audit plan, procedures performed, evidence obtained, and conclusions reached. This documentation is crucial for audit quality and compliance.

Independence and Objectivity: Maintain independence and objectivity throughout the audit process to ensure that the audit is conducted without bias or conflicts of interest. Learning how to keep emotions out of an audit comes with correct training and experience. An auditor who is trained to use “gut-feelings” is trained incorrectly and will continue on the path to fail audits. An SMS enterprise may pass the audit, but the audit itself failed since emotions were the foundation of the audit process. A failed audit is a hazard to aviation safety. An audit failed by an SMS enterprise is the correct path for aviation safety. When an audit is based on emotions, and the auditor is required for the portfolio to fail a certain percentage of their audits, the probability is that they will fail an SMS enterprise who under other circumstances would pass the audit. When emotions are applied, the corrective action plans become hazardous to their operations.   

Material Misstatement Detection: Perform substantive testing to detect material misstatements in their performance statements. This includes testing account balances, e.g. special cause variations, transactions, e.g. data collection, and disclosures, e.g. reports to the accountable executive.

Analytical Procedures: Use analytical procedures (SPC) to identify unusual or unexpected trends, ratios, or fluctuations in raw data collected that may indicate potential issues.

Audit Evidence: Gather sufficient and appropriate audit evidence such as data to support audit conclusions. This includes examining documents, conducting interviews, and performing physical inspections.

Irregularities Detection: Be vigilant for signs of irregularities. Consider factors that could indicate irregularities and conduct additional testing if necessary. Irregularities, or manipulation of an SMS is a simplest task, but is most often done unintentionally by the operators. Within an SMS, an irregularities are often used to embellish hazards and using emotions when identifying a special cause variation.   

Communication: Maintain open and clear communication with management and the audit committee throughout the audit process, especially regarding significant audit findings and issues.

Documentation of Findings: Document any significant findings, including any identified control deficiencies or material misstatements, and communicate them to management in a timely manner.

Audit Reporting: Prepare an audit report that includes the auditor's opinion on the fairness of their SMS and any other required disclosures or findings. Fairness of an SMS is gauged by how SMS principles are applied to regulatory, standards, or SMS policy requirements. E.g. An SMS may apply a stronger leverage to third-party contractors than to their own workers.

Gauging a system involves assessing its performance, effectiveness, and various aspects to determine its current state and potential for changes. The specific method and metrics you use to gauge a system will depend on the nature of the system and your goals. 

When gauging a system, start by clearly defining what you want to achieve by gauging the system. What are the objectives, goals, expectations, and desired outcomes? Understanding the objectives is essential for selecting appropriate gauging methods and metrics.

Determine the key performance indicators (KPIs) or metrics that are most relevant to the objectives. These metrics should be measurable, quantifiable, and directly related to the system's performance. Examples of common metrics include efficiency, accuracy, productivity, cost-effectiveness, and customer satisfaction.

Gather data related to the chosen metrics. Depending on the system, you may collect data through observations, surveys, interviews, experiments, or by analyzing existing records and reports. Ensure that the data is accurate and up-to-date.

Use data analysis techniques such as SPC to evaluate the system's performance based on the selected metrics. This may involve calculating averages, trends, variances, or other relevant statistics. Visualization tools such as charts and graphs can help to present and interpret the data effectively.

Compare your system's performance to established benchmarks or industry standards. Benchmarking can provide valuable insights into how systems perform relative to others in the same domain.

Obtain feedback from the accountable executive, stakeholders, users, or others who have experience with the system. They can provide valuable insights into the system's strengths, weaknesses, and areas for changes.

Based on the data analysis and feedback, identify the strengths and weaknesses of the SMS system. Determine what aspects are performing well and where there is room for changes.

Define specific, measurable, and achievable goals for improving the system. These goals should align with the SMS policy and objectives and focus on strength identified during the gauging process. Weaknesses identified may be used for goalsetting, but focusing on weaknesses does not necessarily, or automatically strengthen a system. A weakness in the SMS is not necessarily a hazard to aviation safety and may be required for the system to function. Overcontrolling by adjusting weaknesses to a strength may cause additional hazards to operations than working with an imperfect system. Conventional wisdom that a weakness of an SMS is shown by quantity of hazard data produced. However, adding irrelevant hazards is the same as overcontrolling the hazard identification process. The regulatory requirement is than an SMS enterprise operates with a process for identifying hazards to aviation safety and for evaluating and managing the associated risks. A hazard which did not affect aviation safety is a non-reportable hazard. E.g. Birds are hazardous to aviation safety, but when the birds are a mile or two away, and they did not an unplanned action by the flight crew, such as reporting to ATC or evasive action, they did not affect safety and therefore not a reportable hazard.

Continuously monitor the SMS system's performance and progress on their path toward the goals. Update metrics and data collection daily to track changes over time.

The gauging process is not a one-time event. It should be an ongoing and iterative process. Periodically revisit objectives, metrics, and goals to adapt to changing circumstances and ensure the system remains effective.

Share the results of the gauging efforts with the accountable executive. Effective communication can foster buy-in and support from workers, customers, users and tenants for change initiatives.

Remember that the specific steps and methods for gauging a system can vary widely depending on the system's complexity and the context in which it operates. Customizing the approach to fit the size and complexity of the SMS system is essential for accurate assessment and meaningful changes.

Follow Ethical Standards: Adhere to ethical standards and professional auditing guidelines, such as those established by relevant auditing standards boards.

Continuous or Continual Learning: Stay updated on changes in auditing standards, regulations, and industry trends to enhance audit quality and effectiveness. Continuous learning is to refresh current knowledge, while continual learning is to add new knowledge to current knowledge.

Quality Control: Ensure that the audit process follows the SMS enterprise’s quality control procedures and standards.

Timeliness: Complete the audit within the established timeline to meet reporting deadlines. An audit should be initiated no later than six months prior to the regulatory audit completion date. The completion date is every three years, counted from the first audit which was due by March 31. 

Feedback and Continuous Improvement: After completing the audit, gather feedback from the audit team to identify areas for improvement in future audits.

Remember that audit procedures may vary depending on the specific audit engagement, so it's essential to tailor these pointers to the size and complexity of the SMS enterprise. Compliance with relevant auditing standards and audit regulations are critical throughout the audit process. Size and complexity of an SMS enterprise is not to ignore, or eliminate regulatory requirements for smaller airport or airlines, but it is to audit to their own established size and complexity daily quality control program. 

When performing an SMS audit there are three key audit pointers, or takeaways that are crucial for the integrity of the audit. 

1) The purpose of an audit is not to fail or pass an SMS enterprise, but to analyze data collected and recorded by an SMS enterprise.

2) Items subject to analytical testing by statistical process control, perform one test only and accept the result.

3) Recommendations by the auditor are not corrective action plan solutions but are recommendations for the SMS enterprise to identify the special cause variations which lead to a regulatory non-compliance, and the special cause variation which lead to a non-conforming operational process. 

OffRoadPilots