1. Sigmoid Function

In Logisttic Regression, the hypothesis is defined as:

where function g is the sigmoid function. The sigmoid function is defined as:

2.Cost function and gradient

The cost function in logistic regression is:

the gradient of the cost is a vector of the same length as θ  where j^th element(for j=0,1,...,n) is defined as follows:

3. Regularized Cost function and gradient

Recall that the regularized cost function in logistic regression is:

The gradient of the cost function is a vector where the j^th element is defined as follows:

for j=0:

for j>=1:

 

Here are the code files:

ex2_data1.txt

 

34.62365962451697,78.0246928153624,030.28671076822607,43.89499752400101,035.84740876993872,72.90219802708364,060.18259938620976,86.30855209546826,179.0327360507101,75.3443764369103,145.08327747668339,56.3163717815305,061.10666453684766,96.51142588489624,175.02474556738889,46.55401354116538,176.09878670226257,87.42056971926803,184.43281996120035,43.53339331072109,195.86155507093572,38.22527805795094,075.01365838958247,30.60326323428011,082.30705337399482,76.48196330235604,169.36458875970939,97.71869196188608,139.53833914367223,76.03681085115882,053.9710521485623,89.20735013750205,169.07014406283025,52.74046973016765,167.94685547711617,46.67857410673128,070.66150955499435,92.92713789364831,176.97878372747498,47.57596364975532,167.37202754570876,42.83843832029179,089.67677575072079,65.79936592745237,150.534788289883,48.85581152764205,034.21206097786789,44.20952859866288,077.9240914545704,68.9723599933059,162.27101367004632,69.95445795447587,180.1901807509566,44.82162893218353,193.114388797442,38.80067033713209,061.83020602312595,50.25610789244621,038.78580379679423,64.99568095539578,061.379289447425,72.80788731317097,185.40451939411645,57.05198397627122,152.10797973193984,63.12762376881715,052.04540476831827,69.43286012045222,140.23689373545111,71.16774802184875,054.63510555424817,52.21388588061123,033.91550010906887,98.86943574220611,064.17698887494485,80.90806058670817,174.78925295941542,41.57341522824434,034.1836400264419,75.2377203360134,083.90239366249155,56.30804621605327,151.54772026906181,46.85629026349976,094.44336776917852,65.56892160559052,182.36875375713919,40.61825515970618,051.04775177128865,45.82270145776001,062.22267576120188,52.06099194836679,077.19303492601364,70.45820000180959,197.77159928000232,86.7278223300282,162.07306379667647,96.76882412413983,191.56497449807442,88.69629254546599,179.94481794066932,74.16311935043758,199.2725269292572,60.99903099844988,190.54671411399852,43.39060180650027,134.52451385320009,60.39634245837173,050.2864961189907,49.80453881323059,049.58667721632031,59.80895099453265,097.64563396007767,68.86157272420604,132.57720016809309,95.59854761387875,074.24869136721598,69.82457122657193,171.79646205863379,78.45356224515052,175.3956114656803,85.75993667331619,135.28611281526193,47.02051394723416,056.25381749711624,39.26147251058019,030.05882244669796,49.59297386723685,044.66826172480893,66.45008614558913,066.56089447242954,41.09209807936973,040.45755098375164,97.53518548909936,149.07256321908844,51.88321182073966,080.27957401466998,92.11606081344084,166.74671856944039,60.99139402740988,132.72283304060323,43.30717306430063,064.0393204150601,78.03168802018232,172.34649422579923,96.22759296761404,160.45788573918959,73.09499809758037,158.84095621726802,75.85844831279042,199.82785779692128,72.36925193383885,147.26426910848174,88.47586499559782,150.45815980285988,75.80985952982456,160.45555629271532,42.50840943572217,082.22666157785568,42.71987853716458,088.9138964166533,69.80378889835472,194.83450672430196,45.69430680250754,167.31925746917527,66.58935317747915,157.23870631569862,59.51428198012956,180.36675600171273,90.96014789746954,168.46852178591112,85.59430710452014,142.0754545384731,78.84478600148043,075.47770200533905,90.42453899753964,178.63542434898018,96.64742716885644,152.34800398794107,60.76950525602592,094.09433112516793,77.15910509073893,190.44855097096364,87.50879176484702,155.48216114069585,35.57070347228866,074.49269241843041,84.84513684930135,189.84580670720979,45.35828361091658,183.48916274498238,48.38028579728175,142.2617008099817,87.10385094025457,199.31500880510394,68.77540947206617,155.34001756003703,64.9319380069486,174.77589300092767,89.52981289513276,1

 

ex2.m

1 %% Machine Learning Online Class - Exercise 2: Logistic Regression  2 %  3 %  Instructions  4 %  ------------  5 %   6 %  This file contains code that helps you get started on the logistic  7 %  regression exercise. You will need to complete the following functions   8 %  in this exericse:  9 % 10 %     sigmoid.m 11 %     costFunction.m 12 %     predict.m 13 %     costFunctionReg.m 14 % 15 %  For this exercise, you will not need to change any code in this file, 16 %  or any other files other than those mentioned above. 17 % 18  19 %% Initialization 20 clear ; close all; clc 21  22 %% Load Data 23 %  The first two columns contains the exam scores and the third column 24 %  contains the label. 25  26 data = load('ex2data1.txt'); 27 X = data(:, [1, 2]); y = data(:, 3); 28  29 %% ==================== Part 1: Plotting ==================== 30 %  We start the exercise by first plotting the data to understand the  31 %  the problem we are working with. 32  33 fprintf(['Plotting data with + indicating (y = 1) examples and o ' ... 34          'indicating (y = 0) examples.\n']); 35  36 plotData(X, y); 37  38 % Put some labels  39 hold on; 40 % Labels and Legend 41 xlabel('Exam 1 score') 42 ylabel('Exam 2 score') 43  44 % Specified in plot order 45 legend('Admitted', 'Not admitted') 46 hold off; 47  48 fprintf('\nProgram paused. Press enter to continue.\n'); 49 pause; 50  51  52 %% ============ Part 2: Compute Cost and Gradient ============ 53 %  In this part of the exercise, you will implement the cost and gradient 54 %  for logistic regression. You neeed to complete the code in  55 %  costFunction.m 56  57 %  Setup the data matrix appropriately, and add ones for the intercept term 58 [m, n] = size(X); 59  60 % Add intercept term to x and X_test 61 X = [ones(m, 1) X]; 62  63 % Initialize fitting parameters 64 initial_theta = zeros(n + 1, 1); 65  66 % Compute and display initial cost and gradient 67 [cost, grad] = costFunction(initial_theta, X, y); 68  69 fprintf('Cost at initial theta (zeros): %f\n', cost); 70 fprintf('Gradient at initial theta (zeros): \n'); 71 fprintf(' %f \n', grad); 72  73 fprintf('\nProgram paused. Press enter to continue.\n'); 74 pause; 75  76  77 %% ============= Part 3: Optimizing using fminunc  ============= 78 %  In this exercise, you will use a built-in function (fminunc) to find the 79 %  optimal parameters theta. 80  81 %  Set options for fminunc 82 options = optimset('GradObj', 'on', 'MaxIter', 400); 83  84 %  Run fminunc to obtain the optimal theta 85 %  This function will return theta and the cost  86 [theta, cost] = ... 87     fminunc(@(t)(costFunction(t, X, y)), initial_theta, options); 88  89 % Print theta to screen 90 fprintf('Cost at theta found by fminunc: %f\n', cost); 91 fprintf('theta: \n'); 92 fprintf(' %f \n', theta); 93  94 % Plot Boundary 95 plotDecisionBoundary(theta, X, y); 96  97 % Put some labels  98 hold on; 99 % Labels and Legend100 xlabel('Exam 1 score')101 ylabel('Exam 2 score')102 103 % Specified in plot order104 legend('Admitted', 'Not admitted')105 hold off;106 107 fprintf('\nProgram paused. Press enter to continue.\n');108 pause;109 110 %% ============== Part 4: Predict and Accuracies ==============111 %  After learning the parameters, you'll like to use it to predict the outcomes112 %  on unseen data. In this part, you will use the logistic regression model113 %  to predict the probability that a student with score 45 on exam 1 and 114 %  score 85 on exam 2 will be admitted.115 %116 %  Furthermore, you will compute the training and test set accuracies of 117 %  our model.118 %119 %  Your task is to complete the code in predict.m120 121 %  Predict probability for a student with score 45 on exam 1 122 %  and score 85 on exam 2 123 124 prob = sigmoid([1 45 85] * theta);125 fprintf(['For a student with scores 45 and 85, we predict an admission ' ...126          'probability of %f\n\n'], prob);127 128 % Compute accuracy on our training set129 p = predict(theta, X);130 131 fprintf('Train Accuracy: %f\n', mean(double(p == y)) * 100);132 133 fprintf('\nProgram paused. Press enter to continue.\n');134 pause;

sigmoid.m

1 function g = sigmoid(z) 2 %SIGMOID Compute sigmoid functoon 3 %   J = SIGMOID(z) computes the sigmoid of z. 4  5 % You need to return the following variables correctly  6 g = zeros(size(z)); 7  8 % ====================== YOUR CODE HERE ====================== 9 % Instructions: Compute the sigmoid of each value of z (z can be a matrix,10 %               vector or scalar).11 12 13 g = 1./(1+exp(-z));14 15 16 % =============================================================17 18 end

costFunction.m

1 function [J, grad] = costFunction(theta, X, y) 2 %COSTFUNCTION Compute cost and gradient for logistic regression 3 %   J = COSTFUNCTION(theta, X, y) computes the cost of using theta as the 4 %   parameter for logistic regression and the gradient of the cost 5 %   w.r.t. to the parameters. 6  7 % Initialize some useful values 8 m = length(y); % number of training examples 9 10 % You need to return the following variables correctly 11 J = 0;12 grad = zeros(size(theta));13 14 % ====================== YOUR CODE HERE ======================15 % Instructions: Compute the cost of a particular choice of theta.16 %               You should set J to the cost.17 %               Compute the partial derivatives and set grad to the partial18 %               derivatives of the cost w.r.t. each parameter in theta19 %20 % Note: grad should have the same dimensions as theta21 %22 hx = sigmoid(X*theta);  % m x 123 J = -1/m*(y'*log(hx)+((1-y)'*log(1-hx)));24 grad = 1/m*X'*(hx-y);25 26 27 28 29 30 31 % =============================================================32 33 end

predict.m

1 function p = predict(theta, X) 2 %PREDICT Predict whether the label is 0 or 1 using learned logistic  3 %regression parameters theta 4 %   p = PREDICT(theta, X) computes the predictions for X using a  5 %   threshold at 0.5 (i.e., if sigmoid(theta'*x) >= 0.5, predict 1) 6  7 m = size(X, 1); % Number of training examples 8  9 % You need to return the following variables correctly10 p = zeros(m, 1);11 12 % ====================== YOUR CODE HERE ======================13 % Instructions: Complete the following code to make predictions using14 %               your learned logistic regression parameters. 15 %               You should set p to a vector of 0's and 1's16 %17 18 p = sigmoid(X*theta)>=0.5;19 20 21 22 23 % =========================================================================24 25 26 end

costFunctionReg.m

1 function [J, grad] = costFunctionReg(theta, X, y, lambda) 2 %COSTFUNCTIONREG Compute cost and gradient for logistic regression with regularization 3 %   J = COSTFUNCTIONREG(theta, X, y, lambda) computes the cost of using 4 %   theta as the parameter for regularized logistic regression and the 5 %   gradient of the cost w.r.t. to the parameters.  6  7 % Initialize some useful values 8 m = length(y); % number of training examples 9 10 % You need to return the following variables correctly 11 J = 0;12 grad = zeros(size(theta));13 14 % ====================== YOUR CODE HERE ======================15 % Instructions: Compute the cost of a particular choice of theta.16 %               You should set J to the cost.17 %               Compute the partial derivatives and set grad to the partial18 %               derivatives of the cost w.r.t. each parameter in theta19 hx = sigmoid(X*theta);20 reg = lambda/(2*m)*sum(theta(2:size(theta),:).^2);21 J = -1/m*(y'*log(hx)+(1-y)'*log(1-hx)) + reg;22 theta(1) = 0;23 grad = 1/m*X'*(hx-y)+lambda/m*theta;24 25 26 % =============================================================27 28 end