Showing posts with label Predictive Modeling. Show all posts
Showing posts with label Predictive Modeling. Show all posts

Tuesday, February 3, 2009

Prediction Model : Step-wise Guide

Data Preparation
o   Business goal.
o   Data sources and merges.
o   Data segmentation-based on product type, other characteristic (utilization or sales) or based on data availability.
o   In sample and out sample data.
o   In case no out sample, creation of Training and Testing datasets from the overall data.

Data Study
o   Contents, Frequency and Univariate procedures.
o   Understanding data distribution (Mean, Median, Mode, Min, Max and Missing).

Data Cleaning
o   Missing Imputation
o   Capping and Flooring (99% or 95% truncations and lower end truncations at 1% (usually)‏)

Derived Variable Creation
o   Representing Character Variables in Numeric Format
o   Flags for Missing etc.
o   Interaction variables using CART/CHAID
o   Transformation (Exponential/Logarithmic)

Variable Reduction
o   Chi-square test & T-test.
o   Cluster Analysis.
o   Factor Analysis.
o   Information Value

Data Segmentation.
o   Model segments based on CART or other characteristics (like-delinquent/non-delinquent).

Rank plots and Transformations.
o   Dummy variable, log, square, square root, exponential, etc. transformations.

Models
o   Linear/Logit /Multinomial Logit etc.
o   Step wise model (backward, forward).

Diagnostic Check.
o   Correlation justifies the Business Sense for all Model variables.
o   All Model variables have same sign for Correlation with DV and Model Coefficient
o   Multicollinearity (Condition Index / VIF)

Model Checks.
o   Correct signs and significance.
o   Actual Vs Predicted (in-sample and out-sample).
o   Lorenz curves (in-sample and out-sample).

Model Comparisons.
o   With Existing model scores
o   Using Gains Table, Gains Chart, Market Richness Chart etc.

 Model Stability Checks
o   Business Checks
o   Statistical Test running CART on Residual DV

Timely Review
o   Review the model and the business environment at times (at least twice a year).  

Wednesday, October 29, 2008

5 Steps to Build a Predictive Model

I keep on getting inquiries about the right way of building predictive models. I am trying to answer those through this post. But before that let me make one thing clear, there is no definite rules to build a Model. Nor there is any good or better methods. It solely depends on the analyst or modeler.
Here is my way. It is just one among the many ways.

1. Modeling Methodology and Entity
As data comes, first thing an analyst need to do is to study the data. Then understand the business problem for which the model is going to be used. These two things lead to the correct definition of Modeling Methodology and Entity.
Modeling Methodology could be choosing one of among the Linear Regression, Logistic Regression and Poisson Regression. And Entity is to define at what level the Model is going to be built.

For banking while accessing risk, typical model build is a risk model. This model is built with Logistic Regression with Good/Bad Dependent Variable. The entity is usually customer. But if you want to see Risk Involved at each subsequent loan that a person has (and believe more the number of loans to a person risk changes), you could go for a Loan Level.

2. Time-line
Not all data received is usable. The timeline needs to be properly finalized by good understanding of data and the business.
Usually for Credit Cards portfolio of banks, the data taken is for 42 months. It includes 24 months of observation period and 18 months of performance period i.e. all cards taken in first 24 months is taken and their subsequent behavior in next 18 months is analyzed in sliding window basis.

This is quite simple. But things are not simple always. If you need to build a Risk Model for US Sub-Prime Market now, this rule might not hold. The business scenario in the last 12 months have changed a lot and in future it is hard to get similar environment. What to do here?

Might be taking a smaller window helps! Or might have to be conservative in building but solve these issues at the implementation time.

3. DV (Dependent Variable or Outcome Variable) Definition
DV definition has its own challenges. Mostly while deciding on methodology and entity, and on timeline there is a definite thought on DV. But there are areas where we need to go much deeper while defining DV.

For credit card, its the risk in 18 months but how do we define risk. It would be two cheque bounces or missed payments; it could be one; It could be crossing the credit limit and not paying thereafter etc. Each of these have business logic behind them.

For building Response Model for a Email Campaign, DV is response but how do you define response? Is it opening email? or replying email? or taking some action based on the email? It could be any.

4. Sampling
This is slightly different from the other steps. I would prefer to use all data that is available and not sample down or up in most scenarios. But there are certain business issues, or data issues or technical BI tools issues which force to do some sampling.

A case of oversampling could be some cases where the data points are too less. And among them too most of the records are good (could be bad too). Say for e.g. we have 1000 records and 990 are goods. The only way to move forward is to increase bads by taking them multiple times.

Other case could be a place where there are 5 million records. Is it necessary to take whole sample?
I would say yes. But there are certain issues of taking 5 million records. I doubt any tool can handle this huge data apart from SAS. And SAS too will take long time to do any simple process and a iterative process like Modeling will just kill any-body's patience.
Solution: Sample down. Either taken random X% or do stratified sampling by taking Y% of Goods and Y% of bads.

5. IDVs (Independent Variables)
IDVs generation is again a art in itself. It is always good to take as much data as you can get, and create as much derived variables as you can from them. This will open up the window for more advanced statistical tests and helps in capturing performance from all angles.

Some of the data sources for IDVs for Financial Clients would be: internal operations data (like sales, call center usage etc.) , demographic data, personal information of customers, economic data, seasonal factors, etc.

The ways of creating derived variables could be:
a. Log of original variable [log(no. of bank card trade)]
b. Taking exponential with e [e(power No. of Bad Loans)]
c. Difference between two similar variables [Loans in last 5 years - Loans in last 1 year]
d. Interaction Variables using CART
e. Inverse Relations [1/No. of Loans]
f. Series Functions e.g. for No. of Loans 1/(Loans in one yr) + 1/(Loans in 2 yrs)^2 + .........
g. Binned Variables using Single Factor Analysis [Age (<18, 18-30,30-65,65+)]

More readings:
1. Modeling on Censored Data
2. Building Predictive Model
3. Time Series Data

I end my note here. It would be interesting to see how you see my note. There might be places where you agree and other areas where you disagree, would like to know them. Please share your comments and feedback through comment or mail.

Tuesday, October 16, 2007

All About Decision Management System

There can be several works in a end to end decisioning system. Decision Management System has to include all those steps. I have tried to summarize them here:
1. Collect and Explore data
2. Understand and Organize Data
3. Design Data Repository and Implement it
4. Build Reporting System (MIS)
5. Analyze Data for hidden insights
6. Perform Predictive Modeling and Segmentation of Portfolio
7. Design Strategy Based on Analysis, Modeling and Segmentation
8. Build and Implement Decision Optimization System
9. Monitor the performance of DO System
10. Fine tune the DO System with feedback from Monitoring System

I am a strong believer of Good Data. Analytics all depends on data, so before performing any analysis on data it has to be ensured that data is Analysis Ready.
Most of the resource intensive work is around data handling. So from points 1 through 4 are related to it.

If some company wants to start using DM system, then its best to follow steps one by one.

#1. Collect and Explore Data: Collect all relevant data either from internal source or from external sources. External sources may be getting information from Data Bureaus or Market Research firms or from partner companies, more the merrier. Then comes exploration of these data collected and finding the necessary ones. All data cannot be analyzed due to resource crunch and it does not make any sense to analyze useless ones.


# 2. Understand and Organize Data: The data in raw form is useless. It needs to be transformed into usable form. So a thorough data study has to be done to understand the business interpretation. Then the knowledge has to be used to organize data for Analytic use. This may include creating derived variables, converting files to usable forms. The usable form may be creating csv files from SQL server in order to convert to SAS (or whichever Analytic Package is used). The Quality Assurance test will need to be transformed in each step, either it is reorganizing data or converting formats.
Also, there might be need to collect some data for future use. Identify such shortage of data.

# 3. Design Data Repository and Implement it: Every time need for Analytics arises performing data works becomes very costly and those data works are mostly common between those functions. So maintaining a good data repository system pays.
Data Repository System should be designed in such a way that all the problems in the first two steps are phased out. The learnings from data exploration and organization can be implemented into an automated system.
If some new data needs to be collected, then build a module in DDR to do that function.


# 4. Build Reporting System (MIS): Data Collection and Maintenance is almost done with the earlier three steps, and now comes reporting. Data in raw form, however good, is hardly interpretable in business. So build a system to report basic insights from data.
Banks may like to know daily cash flow, number of customers visiting daily, rate of increase in credit, number of accounts going charge-off daily etc. There may be hundreds of such metrics that can be directly found out on daily basis and its all repetitive. Such insights when handy can help make many business decisions fairly easy.


# 5. Analyze Data for hidden insights: Not all information data can say is readily available. There might be some hidden reason when some outcome change and it needs to performed on need basis. This might be done routinally in order to avoid sudden accident, or can be done after a alarm.
This might also be needed when a new portfolio program is to be started, or some redesigning is required vis-a-vis competition.


# 6. Perform Predictive Modeling and Segmentation of Portfolio: To take strategic decisions, predictions are always important. It can be best done by segmentation and predictive modeling.

# 7. Design Strategy Based on Analysis, Modeling and Segmentation: Performing the statistical functions does not directly relate to business. It has to be put into business understanding and a strategy needs to be designed. It should include not the Analysis output and Model scores but Business Rules based on those outputs. Each and every decision points need to be evaluated and decisions need to be optimized w.r.t the desired outcome(s).
The desired outcome may be income, revenue, risk, attrition, response etc. based on the business need.

# 8. Build and Implement Decision Optimization System: After the strategy designed, its the time for it to be put in action. In the competitive environment, there might be need to change the strategy quite frequently following the change in market conditions and competition. So each time a different strategy is adopted, putting a new decision system makes it almost impractical in terms of both time and resources. This calls for the need of a Decision Optimization System that allows to change decisions often.
Two of the features of good decision management system can be:
a. creating, changing and removing business rules
b. fine-tuning the operating points for predictive model scores

# 9. Monitor the performance of DO System: Continuous monitoring of DO system is as important as building it. There might be various reasons to do it. The change in market conditions is one, while few business rules and models may have error in them or there might be error in their implementation. But most commonly the need is of change of business need than anything else. There may be many operational issues (resources crunch, lack of expertise etc) while trying new strategy.


# 10. Fine tune and upgrade the DO System with feedback from Monitoring System: The information collected from monitoring system can be used to fine-tune decision points as well as to upgrade the system. No system is perfect and it is hard to find the drawbacks prior to actual implementation, so it is important to follow the path of continuous improvement.

Now, its your turn to say. I have said all I had, and I expect comments here. Please share if you find that I missed to include something or I missed the actual path.

Page Views from May 2007

 

© New Blogger Templates | Webtalks