{"id":1182,"date":"2022-03-14T13:05:39","date_gmt":"2022-03-14T13:05:39","guid":{"rendered":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/?p=1182"},"modified":"2022-03-14T15:22:18","modified_gmt":"2022-03-14T15:22:18","slug":"stochastic-gradient-descent-using-backpropagation-for-deep-reinforcement-learning","status":"publish","type":"post","link":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/2022\/03\/14\/stochastic-gradient-descent-using-backpropagation-for-deep-reinforcement-learning\/","title":{"rendered":"Stochastic gradient descent using backpropagation for deep reinforcement learning"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Learning is an essential part of life, which we do both consciously and subconsciously. From an early age we humans are usually expected to attend school and study. This social custom was introduced hoping that we can start learning about a wide range of topics as early as possible. This form of learning where we intentionally try to change our competences, attitudes and\/or knowledge fits into the category of conscious learning (Simons 2012<\/a>). However, there exists another form of learning. One which we are not fully aware of and \u2018naturally\u2019 happens through the interactions we experience with our environment<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

It is only natural that the ideas behind reinforcement learning are relevant when looking at states that evolve through time. For example, looking at the evolution of rubber tree plants within the Amazon rainforest. In the 1950s Richard Bellman found a way to break these down as to obtain a single value representing the payoff from making a specific decision in terms of both the immediate reward and future discounted reward. These equations are know as the Bellman equations and rely on a so called action-value function. This represents a value of taking a certain action in a specific state. When the number of action and state pairs is very large we can not record each value and instead use a formula to approximate them when needed. This is called functional approximation.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The algorithm used to make the \u2018best\u2019 decision using reinforcement learning tries to minimise a loss function using gradient descent. In essence, by minimising we improve our machine\u2019s ability to make good decisions. This is difficult to do directly, so we will use stochastic gradient descent. Fundamentally, this method approximates our problem and makes it easier to solve, however, when using this method we must compute something called a gradient and these gradients depend on other gradients. This nested structure makes it difficult to estimate. We use a process called backpropagation to solve this problem.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

An example of backpropagation<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Below is a short example from\u00a0Domke (2011)<\/a>\u00a0illustrating why backpropagation works. This example is simple enough so that we
can clearly show each step of how backpropagation works.<\/p>

Given a function f = exp(exp (x) + exp (x)^2 )+sin (exp (x) + exp (x)^2 )<\/span>, we obtain the following expression for its derivative with respect to x<\/span>:<\/p>

\u00a0\\frac{df}{dx}=\\exp{(\\exp{(x)}+\\exp{(x)}^2)}(\\exp{(x)}+2\\exp{(x)}^2)+ \\cos{(\\exp{(x)+\\exp{(x)}^2})(\\exp{(x)}+2\\exp{(x)}^2)}.<\/span><\/p>

However, when the function\u00a0f<\/span><\/span>\u00a0is a lot more difficult to differentiate we can use a different approach. By creating intermediate variables, we can differentiate them step by step and obtain the derivative of the function\u00a0<\/span>f<\/span><\/span>. This is called backpropagation and works as follows. Let:<\/span><\/p>

a\u00a0 = \\exp{(x)} \\\\ b\u00a0 = a^2 \\\\ c\u00a0 = a+b \\\\ d\u00a0 = \\exp{(c)} \\\\ e\u00a0 = \\sin{(c)} \\\\ f\u00a0 = d+e.<\/span><\/p>

Figure 1 gives a visual representation of the relationship between x<\/span> and f<\/span> linked by the intermediate variables created in equation above.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Capture\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Finally, we can simply write the derivative of the intermediate terms using the chain rule work backwards until we obtain the derivative of f with respect to x<\/span>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Capture2\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Learning is an essential part of life, which we do both consciously and subconsciously. From an early age we humans are usually expected to attend school and study. This social custom was introduced hoping that we can start learning about a wide range of topics as early as possible. This form of learning where we […]<\/p>\n","protected":false},"author":40,"featured_media":1260,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ocean_post_layout":"full-width","ocean_both_sidebars_style":"","ocean_both_sidebars_content_width":0,"ocean_both_sidebars_sidebars_width":0,"ocean_sidebar":"0","ocean_second_sidebar":"0","ocean_disable_margins":"enable","ocean_add_body_class":"","ocean_shortcode_before_top_bar":"","ocean_shortcode_after_top_bar":"","ocean_shortcode_before_header":"","ocean_shortcode_after_header":"","ocean_has_shortcode":"","ocean_shortcode_after_title":"","ocean_shortcode_before_footer_widgets":"","ocean_shortcode_after_footer_widgets":"","ocean_shortcode_before_footer_bottom":"","ocean_shortcode_after_footer_bottom":"","ocean_display_top_bar":"default","ocean_display_header":"default","ocean_header_style":"","ocean_center_header_left_menu":"0","ocean_custom_header_template":"0","ocean_custom_logo":0,"ocean_custom_retina_logo":0,"ocean_custom_logo_max_width":0,"ocean_custom_logo_tablet_max_width":0,"ocean_custom_logo_mobile_max_width":0,"ocean_custom_logo_max_height":0,"ocean_custom_logo_tablet_max_height":0,"ocean_custom_logo_mobile_max_height":0,"ocean_header_custom_menu":"0","ocean_menu_typo_font_family":"0","ocean_menu_typo_font_subset":"","ocean_menu_typo_font_size":0,"ocean_menu_typo_font_size_tablet":0,"ocean_menu_typo_font_size_mobile":0,"ocean_menu_typo_font_size_unit":"px","ocean_menu_typo_font_weight":"","ocean_menu_typo_font_weight_tablet":"","ocean_menu_typo_font_weight_mobile":"","ocean_menu_typo_transform":"","ocean_menu_typo_transform_tablet":"","ocean_menu_typo_transform_mobile":"","ocean_menu_typo_line_height":0,"ocean_menu_typo_line_height_tablet":0,"ocean_menu_typo_line_height_mobile":0,"ocean_menu_typo_line_height_unit":"","ocean_menu_typo_spacing":0,"ocean_menu_typo_spacing_tablet":0,"ocean_menu_typo_spacing_mobile":0,"ocean_menu_typo_spacing_unit":"","ocean_menu_link_color":"","ocean_menu_link_color_hover":"","ocean_menu_link_color_active":"","ocean_menu_link_background":"","ocean_menu_link_hover_background":"","ocean_menu_link_active_background":"","ocean_menu_social_links_bg":"","ocean_menu_social_hover_links_bg":"","ocean_menu_social_links_color":"","ocean_menu_social_hover_links_color":"","ocean_disable_title":"default","ocean_disable_heading":"default","ocean_post_title":"","ocean_post_subheading":"","ocean_post_title_style":"","ocean_post_title_background_color":"","ocean_post_title_background":0,"ocean_post_title_bg_image_position":"","ocean_post_title_bg_image_attachment":"","ocean_post_title_bg_image_repeat":"","ocean_post_title_bg_image_size":"","ocean_post_title_height":0,"ocean_post_title_bg_overlay":0.5,"ocean_post_title_bg_overlay_color":"","ocean_disable_breadcrumbs":"default","ocean_breadcrumbs_color":"","ocean_breadcrumbs_separator_color":"","ocean_breadcrumbs_links_color":"","ocean_breadcrumbs_links_hover_color":"","ocean_display_footer_widgets":"default","ocean_display_footer_bottom":"default","ocean_custom_footer_template":"0","slim_seo":{"title":"Stochastic gradient descent using backpropagation for deep reinforcement learning - Thomas Newman","description":"Learning is an essential part of life, which we do both consciously and subconsciously. From an early age we humans are usually expected to attend school and st"},"ocean_post_oembed":"","ocean_post_self_hosted_media":"","ocean_post_video_embed":"","ocean_link_format":"","ocean_link_format_target":"self","ocean_quote_format":"","ocean_quote_format_link":"post","ocean_gallery_link_images":"off","ocean_gallery_id":[],"footnotes":""},"categories":[11],"tags":[],"class_list":["post-1182","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-statistics","entry","has-media"],"_links":{"self":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/posts\/1182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/comments?post=1182"}],"version-history":[{"count":74,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/posts\/1182\/revisions"}],"predecessor-version":[{"id":1287,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/posts\/1182\/revisions\/1287"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/media\/1260"}],"wp:attachment":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/media?parent=1182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/categories?post=1182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/wp-json\/wp\/v2\/tags?post=1182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}