Part of Toolkit for the Economic Evaluation of World Bank Transport Projects
(Institute for Transport Studies, University of Leeds, 2003)
The demand forecasts form a key input to the economic appraisal. As such any errors present within the demand forecasts will undermine the reliability of the economic appraisal. The minimisation of demand forecasting errors is therefore important in the delivery of a robust appraisal.
In this note we address this issue by introducing the key issues and error types present within demand forecasts (Section 1). Following that introductory section the error types are described in more detail: measurement error (Section 2), model specification error (Section 3) and External or Exogenous Errors (Section 4). The final section presents a discussion on how to manage demand forecasting errors (Section 5).
Any errors that are present within the demand forecasts will impact on the reliability of the economic appraisal. The dependency of the economic appraisal on the demand forecasts arises for the reasons set out below.
- A transport investment will alter travel patterns and an understanding of this change (i.e. a demand forecast) is needed to enable an appraisal to take place.
- Demand forecasts provide estimates of key inputs to the calculation of user benefits. For example traffic flow measures (such as vehicles, passengers, tonnes of freight), as well as travel times and vehicle speeds are required for the calculation of travel time and operating cost benefits.
- Economic appraisal occurs over the life of a project (e.g. 30 years – see Project Life note). Estimates of the manner that the user benefits vary over time are therefore required. Demand forecasts for future years are therefore a fundamental input to an economic appraisal.
Demand forecasts are an estimate of the future based upon the existing situation. The existing situation is described using travel demand volumes and patterns as well as supply conditions (e.g. travel time). The demand model itself contains some form of mathematical relationship that will relate travel demand with supply conditions and external or exogenous factors (e.g. GDP growth). This relationship will contain behavioural parameters that reflect the importance and responsiveness of demand to the factor in question (e.g. travel time). A result of this process is that three types of forecasting error can exist; measurement (or data) errors, model specification errors and external errors.
Measurement or data errors occur as a complete dataset of travel demand patterns, supply conditions and behavioural parameters is fundamental to the accurate description of the existing situation but very costly to obtain. In the majority of situations the dataset available gives only a partial coverage, is old or is a mixture of the two. Most demand forecasting models therefore rely on estimation techniques and professional judgement for the development of the base year model (which represents the existing situation) and for the derivation of variables that interface the demand forecasts with the economic evaluation.
Measurement or data errors are encountered and are important for the following data types and reasons:
DEMAND (TRAFFIC PATTERNS AND VOLUMES)
· Road traffic flows, though these can be relatively easily measured (on a single day) they are subject to hourly and daily as well as seasonal variation.
· Public transport passenger volumes can be more difficult to obtain than data on the number of vehicles (e.g. buses) due to widespread use of informal bus stops (in less regulated countries), commercial confidentiality and the difficulty associated with estimating passenger volumes without boarding the vehicle. Again such flows vary daily and seasonally.
· Origin-destination movements of travellers are difficult to obtain without interviewing, which is costly. Often partial datasets are used as a basis for estimation.
· Freight volumes/tonnage/values can be difficult to obtain due to commercial confidentiality
SUPPLY CHARACTERISTICS
· Travel times for public transport services can be extracted from timetables if they are reliable. For private road traffic travel time surveys are needed and for unreliable public transport services travel time surveys would also be needed.
· Journey distances are very important and can often be extracted from geographic information systems or measured from maps. Actual journey distances are almost always larger than the crow fly distance.
· Public transport service frequencies and routes/lines are difficult to obtain without surveying in countries or areas with little regulation or where no timetable exists (i.e. a demand responsive or sporadic service exists).
EVALUATION PARAMETERS
· Annualisation factors and factoring single day observations to an average daily level require continuous observed data throughout the calendar year. Errors in this data can have a very significant impact on the validity of the results.
· Working time and Non-working time split requires interview surveys which are costly
GENERAL
· General data entry errors (e.g. transcription errors, transposition errors, etc.)
The demand forecasting model will generally contain a mathematical relationship and have a spatial context. This relationship will utilise the demand and supply inputs described above to output demand forecasts at most probably a spatially disaggregated level. Some demand models, particularly road traffic models, may also output the change in supply conditions (e.g. change in travel times in response to the increase in congestion). The mis-specification of any of the relationships contained within the demand forecasting model will lead to errors within the demand forecasts.
Key aspects of the demand forecasts where model specification errors may impact on an economic assessment are set out below.
Model area: the model should encompass the area over which all the economic benefits and costs of the project will be felt. Often a project may have cross-border impacts, for example in the case of project aimed at stimulating international trade or in the case of a large urban city located near a national boundary. In such instances it is critical that the model area should encompass all parts of the transport network affected by the project no matter which country the network lies in. The model area should not therefore be artificially restricted to respect the borders of one country.
Model segmentation: the model should be segmented to reflect the following characteristics of the study area and proposed transport investment:
· Where user benefits are expected to vary temporally (e.g. either hourly or seasonally). Thus a model of a congested urban area should be segmented to reflect the morning peak, the inter-peak and the evening peak, whilst a model of a rural area may need to be segmented to reflect the wet season and the dry season should say travel times vary significantly between the two.
· Where different growth rates for vehicle classes that accrue different levels of user benefit (e.g. cars and trucks) are expected. In such situations the vehicle classes should be explicitly differentiated between within the model.
· Where significant variations in behaviour could be expected from different population groups. In the majority of situations this is restricted to the disaggregation between commercial or freight traffic and passenger carrying traffic, however, in situations where some form of fiscal regime is envisaged (e.g. road tolls, public transport fares) segmentation by in-working time and non-working time travelling groups and/or by income banding will be needed.
Omission or mis-specification of key variables: the demand forecasting model may fail to take account of the impact of key explanatory variables (e.g. income or walking time). Another source of error could be that the effect of an explanatory variable has been mis-specified, for example the balance between travel time and distance in determining route choice of road vehicles or the demand elasticities have been wrongly measured. The effects of key explanatory variables should be calibrated to the local area wherever possible, and where defaults are used these should be based on behavioural and not resource values (e.g. value of time).
Future year changes in behavioural parameters: such changes will occur with income growth and changes in the real prices of commodities (e.g. fuel price). Such changes should be reflected in the modelling.
Transferability: a model successfully developed in one area at a certain point of time may not be transferable to another area and/or point of time.
Aggregation error and the scale factor problem: Models, e.g. the logit are often calibrated with disaggregate data but applied with aggregate data. This will lead to bias as the average of a set of non-linear functions will not be the same as a non-linear function of a set of averages. Models based on stated preference data and the logit model may be affected by this technical problem, biasing absolute forecasts
External or exogenous errors are associated with the external inputs or assumptions that underpin the demand forecasting model. They include assumptions regarding external factors (e.g. GDP or income growth), errors in the planning assumptions, ramping up effects, definition of the Do Minimum and Do Something and their interaction with other schemes under consideration as well as the competitive response of other transport operators. Often these inputs are difficult to forecast and as a result they are treated as external within the demand forecasting model. Historically errors in these variables or assumptions have formed the main reason that demand forecasts in the planning stage have differed from out turn flows (vehicle volumes, passenger volumes or freight tonnes) (see Mackie and Preston (1996). A description of the main sources of external or exogenous errors is set out below.
Errors in planning assumptions: many schemes are dependent upon planning decisions. This could include approval for the development of land by industry, which may be finally rejected, or conversely the transport investment maybe developed to serve a community which for a variety of reasons declines.
External factors incorrectly forecast: most transport forecasts are heavily dependent upon the forecast of external factors, principally GDP or income growth, population, economic activity and car ownership. Errors in such assumptions can have a significant impact on growth potential.
Future year transport inputs: this is most pertinent to public transport schemes particularly new ones (e.g. light rail), where estimates of future fare levels, service frequencies and travel times have to be made. Should these estimates differ from those that actually occur a significant difference in the demand forecast may occur. With respect to road schemes, particularly if the HDM software suite is used, the definition of the supply side often forms part of the demand model through the use of speed flow curves and the interaction with the maintenance strategy and the assumptions made relate to the functional form of the speed/flow relationships amongst others.
Ramp Up effects: there are three main effects; firstly disruption caused during the construction of a scheme may mean that some of the base or inscope traffic that would otherwise have transferred to the scheme does not as they find alternative means of transport. Secondly, demand requires time to build up, and such a time lag is often neglected. The third effect is particularly pertinent to new technology (e.g. new trains) where technology related unreliability problems in the early years can prevent out turn flows meeting the demand forecasts.
Interactions with other transport operators and services: the reaction of rival transport operators to a transport investment can in certain circumstances be quite strong. Examples include the ability of the road haulage industry (even if fragmented) to react to a new freight railway, ferry operators reacting to a tolled fixed link crossing. Any assumption that the new service may therefore capture all of the existing market may therefore well be an overestimation. With respect to roads traffic re-assignment (i.e. the interaction between roads) is a significant response and should always be accounted for within the demand forecasting process.
Do Minimum and Do Something: the definition of Do Minimum and Do Something scenarios is described in the Framework [Link] and the role of alternative scenarios is discussed in the World Bank’s Handbook on Economic Analysis of Investment Operations. The correct description and forecasting of the Do Minimum and the alternative Do Something scenarios is important. For example, with respect to the Do Minimum it is important to ensure that future maintenance profile and costs are correctly defined or that rolling stock refurbishment or replacement costs are incorporated into the Do Minimum (see Note Treatment of Maintenance [Link] and Note Sources of Operating Cost [Link]). The Do Something options must be realistic and comparable alternatives to both each other and the problems that the investment will address.
Transport system effects: demand forecasts for a scheme can be sensitive to other proposals for the transport system. Other transport proposals can be divided into competing and complementary projects. Competing projects will extract demand from the scheme under consideration. For example, the demand for scheme that serves railway freight may be sensitive to the gradual upgrading over a 20 year period of the road network, or the demand forecasts for one port will be sensitive to proposals at other competing ports. Complementary projects on the other hand may increase the demand for a transport scheme, for example the demand for a road may increase if a complete route upgrade (at a national level occurred). Conversely assuming the national route upgrade will go ahead come what may will probably overstate the potential demand forecasts (see errors in planning assumptions above).
Future Years Modelled: external errors can occur in the estimate of economic user benefit as a result of interpolation of benefit from a limited number of demand forecasts. To avoid this where user benefits are expected to increase exponentially with growth (e.g. in congested urban areas) demand forecasts should be produced at five year increments as input to the economic appraisal (e.g. for Opening Year (OY), OY+5, OY+10 and OY+15). In uncongested environments demand forecasts for the OY and OY+15 would normally be sufficient. Post OY+20 population and land use forecasts (the demand drivers) are less reliable and travel demand growth should be frozen.
Demand forecasting errors will affect the economic appraisal, however, it is possible to minimise their impact through careful consideration of the demand forecasting methodology and the validation of the model. Such a consideration will minimise measurement or data errors and model specification errors. External or exogenous errors are more difficult to minimise, however, sense checks on the main components of benefit and on key aspects of the underlying assumptions are very worthwhile. Once minimised the impact of the demand forecasting errors (along with other errors) can be assessed using the approaches set out in the Note Risk and Uncertainty Analysis [Link].
Model Validation: a thorough process of model validation is fundamental to the minimisation of demand forecasting errors, particularly measurement/data errors and model specification errors. It should be demonstrated that the demand forecasting model reproduces existing year behavioural patterns and that all the inputs to the transport model have been checked (e.g. logic checks on link lengths). If the model is incremental in nature, that is the behavioural responses are not validated through a reproduction of the existing situation, the model should be sensitivity tested to ensure that it produces sensible results. Sensitivity of modelled behavioural responses can be compared to those exhibited in other regions or other countries. Ideally, however, behavioural responses should be compared to those surveyed as part of the study (e.g. through stated preference surveys).
Disaggregation of Demand Forecasts: the demand forecasts should be disaggregated between the demand that currently exists on the route (base load), the demand that exists but currently uses alternative routes (diverted load) and the demand that has completely new or has been extracted from competing modes (generated load). Table 1 in the Note Treatment of Induced Traffic [Link] defines more precisely the terms base, re-assigned and generated traffic. A consideration of the following issues for each of these demand types will assist in minimising the demand forecast errors. Obviously for a scheme that derives the majority of its benefit from improvements to base traffic the focus of the checks should be on the adequate representation of the base demand, whilst for a scheme that derives the majority of its benefit from generated traffic the focus of the checks should be on ensuring the generated traffic forecasts are defensible.
· BASE TRAFFIC –
(i) Model base year flows should be representative of those that actual occur. Suitable observed data should exist to disaggregate the user benefit between vehicle groups and users (e.g. private and commercial traffic, in work time travellers to non-work time travellers).
(ii) Background growth forecasts that differ from time trend data should be fully explained and defended (e.g. rail patronage market share is forecast to grow despite significant historic decline in market share, or road traffic is forecast to grow at a much higher rate than previous growth (correlated against GDP growth) would suggest).
· RE-ASSIGNED TRAFFIC – market share of new infrastructure is reasonable and existing facilities still maintain some market share (e.g. for a road proposal a competing road route still attracts road traffic or for a port proposal a competing port still maintains some market share of freight market)
· INDUCED TRAFFIC –
(i) Is included when aim of project is to stimulate economic development in certain regions or open access to new markets
(ii) The market share of the project is reasonable in relation to competing modes (e.g. a new rail freight route will not extract all the demand from trucks nor will coastal shipping fully replace either trucks or rail)
(iii) The growth in generated traffic over time does not contradict or work in the opposite direction to background growth (i.e. should growth in generated traffic exhibit a very different growth rate to background growth this should be explained and defended)
Components of benefit validation: not only should the ability of the model to accurately reflect demand have been demonstrated (see above) but it’s ability to properly reflect the manner that that benefit accrues should have been demonstrated. Thus for example should user benefits accrue principally through travel times the reasonableness of the model at representing existing travel times should have been demonstrated and an appropriate mechanism for the calculation of future year travel times should be incorporated. Another example would be that a scheme within a congested urban area would be expected to generate significant benefits within the peak period and as such the peak periods should be explicitly represented within the model.
Demand forecasts and User Benefit Interface: Annualisation factors that convert the demand forecasts to an annual basis are absolutely critical to the level of error that is present in the economic appraisal. As these factors are often applied in a multiplicative manner a 10% error in the factor implies a 10% error in the user benefit calculation. The basis for the annualisation factors should therefore be clearly set out and reviewed. Similarly the manner that user benefits have been interpolated or extrapolated over the project life should be considered in detail particularly in urban areas (see future years modelled above).
External Assumptions: the rationale and basis for all external assumptions should be reviewed and checked for appropriateness. In particular the following features should be considered:
· GDP and income growth forecasts
· Population forecasts
· Car ownership forecasts
· Land use planning – the location size and likelihood of land use developments occurring or conversely reducing in size. Additionally, the treatment of development traffic with respect to general background traffic growth should be considered (i.e. has the development traffic been considered to be part of or additional to the background growth)
· Future year supply side inputs to the model such as public transport fares, service frequencies etc., particularly for new schemes
· Interaction and competition with other transport operators (particularly between modes)
· Competing and complementary schemes
· Ramp up effects
[1] Mackie, P.J. and Preston J. (1998) Twenty one sources of error and bias in transport project appraisal. Transport Policy 5, 1-7
[2] World Bank (1998), Handbook on Economic Analysis of Investment Operations. Washington, DC: The World Bank. Technical Appendix. Available online at:
http://www.worldbank.org/education/economicsed/project/projwork/Handbook.pdf
[3] Ortuzar, J. de D. and Willumsen, L.G. (1994) Modelling Transport. (second edition) John Wiley and Sons, Chichester, UK.
[4] World Bank Roads and Highways Planning Demand Forecasting (includes links) [Available online at:
http://www.worldbank.org/transport/roads/pl_sect6.htm#demand]
[5] Highways Agency, UK (1991), Traffic Appraisal Manual, DMRB Volume 12.1.1. Her Majesty’s Stationary Office, UK [Available online at:
http://www.official-documents.co.uk/document/deps/ha/dmrb/index.htm]
[6] Highways Agency, UK (1996), Traffic Appraisal in Urban Areas, DMRB Volume 12.2.1. Her Majesty’s Stationary Office, UK [Available online at:
http://www.official-documents.co.uk/document/deps/ha/dmrb/index.htm]
[7] Highways Agency, UK (2003), Guidance on Induced Traffic, DMRB Volume 12.2.2. Her Majesty’s Stationary Office, UK [Available online at:
http://www.official-documents.co.uk/document/deps/ha/dmrb/index.htm]
[8] Department for Transport, UK (2000), Guidance on the Methodology for Multi-Modal Studies (GOMMS), Volume 2, Appendix A (Transport Models) [Available online at http://www.dft.gov.uk/stellent/groups/dft_control/documents/contentservertemplate/dft_index.hcst?n=7923&l=3]