Practical SMS Evaluation
An SMS enterprise conducts has established regular self evaluation processes of its safety management system (SMS). The most effective self evaluation method is to link daily operational processes to regulatory requirements, standards, and their SMS policy in a daily quality control system. A governing principle of a successful SMS is to collect operational data, e.g. daily, hourly or by tasks, depending on size and complexity, for process analysis. Evaluating a safety management system is ongoing self-evaluation of the entire system. Airline operators are using this process in their Flight Data Monitoring system (FDM), where flight operational parameters are monitored, and data collected for selected parameters.
A flight data monitoring system is different that the aircraft “black box”, since it is a system for airlines to review collected aircraft parameter data in real time, or daily. Generally speaking, the two reasons to collect FDM data points frequently are to know what is happening in real time, or close to real time, and to manage manpower workload. It is more effective to work with 25 datapoints daily for an aircraft, than it is to catch up with 25,000 data points annually. A self evaluation process of the safety management system work by these same principles, which is the collection of daily data points, or more often if required by size and complexity of operations.
A self evaluation process is built to include the required items and tasks of a safety management system. Items and tasks to include are their safety policy, process for setting goals and for measuring the attainment of those goals, process for identifying hazards and for evaluating and managing the associated risks, a process for ensuring that personnel are trained and competent to perform their duties, a process for the internal reporting and analyzing of hazards, incidents and accidents and for taking corrective actions, a document containing (SMS Manual) all safety management system processes and a process for making personnel aware of their responsibilities with respect to them, a quality assurance program, a process for conducting periodic reviews or audits of the safety management system and reviews or audits, for cause, of the safety management system, and any additional requirements for the safety management system that are required by the regulations.
A self evaluation of the safety management system includes an evaluation of the quality assurance program. An evaluation of a quality assurance program includes an evaluation of regular scheduled audits or audits for cause of airport activities. An evaluation also evaluates how findings resulting from the quality assurance program are distributed to the appropriate manager for corrective action and follow-up. There are two options for the design of an audit system. One complete audit may be carried out every three years, calculated from the initial audit, or a series of audits conducted at intervals set out in the airport operations manual. When using the Daily Rundown FDM process at airports, data points are collected daily for ongoing self-evaluation of the safety management system.
An evaluation of a safety management system must include the position managing the SMS. This position may be named differently, such as the SMS manager position, safety manager position, or safety officer position. The aviation industry is moving in the direction of autonomous aircraft and vehicles. At the time of writing there are no autonomous safety management system, and every SMS enterprise has a person as their SMS manager. Without an evaluation of this position, the SMS manager position, it is unknown how successful an SMS enterprise is operating.
An evaluation of the SMS manager position includes the reporting system, hazard identification, risk management, investigation, analyzes and root cause of all hazards, incidents, and accidents. The evaluation includes an evaluation of their systems to monitor and analyze trends in hazards, incidents, and accidents, monitor and evaluate the results of corrective actions, monitor the concerns of the civil aviation industry in respect of safety and their perceived effect on their operations and the adequacy of the training required.
Assessment typically refers to the process of gathering information and evidence to make informed decisions about an individual, a program, or a system. Assessment can be formative (ongoing, to inform instruction) or summative (final, to make judgments about effectiveness).
Evaluation involves making judgments about the quality, value, or effectiveness of something. Evaluation often comes at the end of a process and involves assessing the overall impact or success of an activity, program, or project.
Assessment often focuses on specific skills, knowledge, or outcomes. It may involve the measurement of individual performance or the effectiveness of a specific aspect of a program.
Evaluation examines the broader picture, considering the overall effectiveness, efficiency, and relevance of a program or project. It may involve multiple components and assess how well the entire initiative met its goals.
Assessment is ongoing and occur at various stages of a process. It may be used to provide feedback and guide improvement.
Evaluation takes place after the completion of a program or project. It involves looking back at the entire process to determine its success and identify areas for improvement. Using the Daily Rundown system as their FDM, an SMS enterprise is evaluating processes upon their completion.
Assessment informs decisions about individual performance, instructional strategies, or the need for adjustments during an ongoing process.
Evaluation informs decisions about the overall success or failure of a program, project, or initiative. It may influence future planning, funding decisions, or resource allocation.
It is important to note that these distinctions are not always rigid, and the terms may be used differently in various fields or contexts. In some cases, people use assessment and evaluation interchangeably, and the specific meanings can depend on the goals and perspectives of organizations involved.
A practical SMS evaluation involves a goalsetting plan. Without a plan, goals are only wishes and dreams. The 14-step goalsetting plan is a practical plan to implement into any SMS.
Clearly articulate your long-term vision and ultimate objectives.
Determine the order of importance for your goals and focus on the most critical ones.
Divide each goal into smaller, manageable milestones to track progress more effectively.
Anticipate potential challenges and obstacles that may hinder your progress.
Create detailed plans outlining the specific actions needed to achieve each milestone.
Identify the resources (financial, time, personnel) required to execute your action plans.
Set deadlines for each milestone to create a sense of urgency and accountability.
Regularly review your progress, identify what's working and what isn't, and make adjustments as needed.
Acknowledge and celebrate small victories along the way to stay motivated.
Stay Flexible:
Be open to adjusting your goals and strategies based on changing circumstances or new information.
Seek Feedback:
Solicit input from mentors, peers, or experts to gain valuable perspectives and insights.
Reflect and Learn:
Reflect on your experiences, learn from both successes and failures, and apply those lessons to future endeavors.
Reassess and Set New Goals:
Periodically reassess your long-term vision, adjust goals as needed, and set new objectives based on your evolving priorities.
An SMS enterprise should customize a 14-step goalsetting plan within each subheading to align with specific circumstances and preferences as an airport or airline operator.
When conducting an evaluation of the SMS, it is just as important to evaluate non-written processes as it is to evaluate written processes. Remember, not everything can be written down to explain a person how to do their job. There will be times when non-written, but still acceptable work practices will be applied. A trap for accountable executives to avoid is if something should go wrong when unwritten processes are used, is to restrain themselves from assigning an unwritten task as the root cause. In the absence of the written process text, a root cause decision can only be based on opinions and emotions.
Believing that nobody engages in non-written processes can pose several dangers for a Chief Executive Officer (CEO) and the organization as a whole.
Non-written processes often involve informal communication, ad-hoc decision-making, and undocumented workflows. If the CEO dismisses the existence of such processes, they may have an incomplete understanding of how work is actually done within the organization.
Non-written processes can sometimes highlight areas where personnel have found more efficient or effective ways of doing their work. Ignoring these processes may mean missed opportunities to identify and implement improvements.
Innovation often stems from informal discussions, brainstorming sessions, and creative collaboration. If the CEO dismisses the importance of non-written processes, it may hinder the organization's ability to foster a culture of innovation.
Personnel may feel undervalued or ignored if their informal contributions are not recognized. This can lead to decreased morale and engagement, potentially impacting productivity and overall organizational performance.
Non-written processes sometimes involve compliance-related activities that may not be documented in formal procedures. Ignoring these processes can lead to compliance issues and legal risks for the organization.
Non-written processes often provide the flexibility, or resilience needed to promptly adapt to changing, or unforeseen circumstances. Rigid adherence to written processes alone may hinder an SMS enterprise to respond effectively to unforeseen challenges. Unwritten processes or tasks are special cause variations and the result of lack of organizational preparation of common cause variations.
Succession planning often relies on a deep understanding of how work is done within an organization. Disregarding non-written processes may make it challenging to identify and groom potential successors who understand the nuances of the business.
To address these risks, It is crucial for CEOs to recognize the value of both written and non-written processes. A balanced approach that acknowledges the importance of informal communication, creativity, and adaptability can contribute to a more resilient and dynamic organizational culture. Both written and unwritten processes must be accepted by an SMS enterprise’s SMS policy and their non-punitive reporting policy.
An SMS evaluation needs to come with a practical approach. Before assigning someone a practical approach, it must also be known and understood what an impractical approach is. Maintaining an impractical approach to an evaluation means adhering to methods, criteria, or perspectives that are unrealistic or not feasible in the context of the evaluation process. This could involve using criteria that are overly idealistic, difficult to implement, or not relevant to the goals and constraints of the situation.
For example, if an SMS enterprise is evaluating a project's success and insist on criteria that are unattainable or do not align with the project's objectives, would be maintaining an impractical approach. This could lead to biased or inaccurate assessments, as the criteria are not grounded in the practical realities of the situation.
It is essential in evaluations to strike a balance between setting high standards and being realistic about what can be achieved given the resources, constraints, and objectives of the project or process being evaluated. An impractical approach can hinder effective decision-making and problem-solving, as it may lead to conclusions that are disconnected from the practical realities of the situation.
Maintaining a practical approach to an evaluation means adopting a realistic and pragmatic mindset when assessing a situation, project, or problem. It involves focusing on tangible and achievable goals, considering real-world constraints, and utilizing practical solutions. Here are some key aspects of maintaining a practical approach to evaluation:
A practical approach is to establish realistic goals and define achievable and measurable goals that align with the overall objectives, and considering resources, time, and constraints involved in achieving those goals.
Data-Driven Decisionmaking is to base evaluations on reliable and relevant data rather than solely on assumptions or personal opinions. Use quantitative and qualitative data to inform decisions and draw meaningful conclusions.
Cost-Benefit Analysis is to evaluate the costs and benefits associated with different courses of action. Consider the return on investment and assess whether the benefits justify the resources expended. This is crucial for a successful SMS. Without a cost benefit analysis, the path forward is unplanned.
A cost-benefit analysis (CBA) is a systematic process for assessing and evaluating the potential advantages (benefits) and disadvantages (costs) of a decision, project, or policy. It is commonly used in business, economics, and public policy to make informed choices by comparing the expected benefits with the expected costs. The goal of a cost-benefit analysis is to determine whether the benefits outweigh the costs and whether the investment or decision is economically justified.
Clearly define the decision, project, or policy under consideration and establish the boundaries of the analysis. Identify and list all the potential costs and benefits associated with the decision or project. These can include direct costs, indirect costs, tangible benefits, and intangible benefits. Assign a monetary value to each cost and benefit. This step can be challenging for intangible factors, such as environmental impact or improved quality of life, but it is essential for comparing all elements on a common scale. Account for the time value of money by discounting future costs and benefits to their present value. This involves adjusting future values to reflect their current worth. Determine the net present value by subtracting the total discounted costs from the total discounted benefits. A positive NPV indicates that the benefits exceed the costs. Assess the impact of uncertainties or changes in key variables on the results. This helps to understand how sensitive the analysis is to variations in assumptions. Establish decision criteria, such as a minimum acceptable rate of return or a specific threshold for the benefit-cost ratio, to guide decision-making, which must be linked to the SMS policy. Based on the results of the analysis and the decision criteria, make an informed decision about whether to proceed with the project, policy, or investment.
Cost-benefit analysis is an invaluable tool for the accountable executive because it provides a structured framework for evaluating options and making choices that maximize overall welfare or value.
Context Awareness is to understand the specific context in which the evaluation is taking place. Consider external factors, cultural nuances, and the environment that may impact the outcomes.
Practical Recommendations is to provide actionable and implementable recommendations. Consider the feasibility and practicality of suggested solutions within the given context.
Stakeholder Involvement is to engage relevant stakeholders in the evaluation process to gather diverse perspectives. Take into account the needs and concerns of those who are directly affected by the evaluation outcomes.
Communication is to clearly communicate evaluation findings and recommendations in a way that is easily understood by stakeholders. Highlight practical implications and potential actions that can be taken.
Learn from Experience is to reflect on past experiences and learn from both successes and failures. Use feedback and lessons learned to continuously improve the evaluation process.
Incorporating these elements into the SMS evaluation process, personnel and the accountable executive can ensure that their assessments are grounded in practicality, facilitating effective decision-making and implementation of recommendations.
A practical SMS evaluation is to evaluate SMS by monitoring processes, procedures, and acceptable work practices. When unwritten processes are identified as special cause variations, conduct a root cause analysis, and implement practical tasks into the written process text. It is crucial for success to accept that it is impossible for all personnel, including the AE, do not recall the text, or steps in a process. The last task completed must therefore act as a trigger to the next step of the process.
Practical may not always be simple, or easy, but it is practical. Ultimately, the level of simplicity in practical solutions depends on factors such as the nature of the problem, the available resources, the target audience, and the goals of the solution. It is essential to strike a balance between simplicity and complexity, choosing an approach that effectively addresses the problem at hand while being practical and manageable.
OffRoadPilots
Comments
Post a Comment